Is agile software development a living design process?

I recently wrote a post outlining some principles for doing Living Design Process, a concept developed by permaculturist, Dan Palmer. These were inspired by my experience with a house renovation and permaculture-inspired garden project.

As I was reading back over the post, I realised that these principles were familiar to me from my work in agile software development.

In theory, agile software development should be an example of a living design process. In practice, it’s not. I’ll use the 7 principles from my blog post to explain why.

Principle 1: Embed yourself in the context

We do this pretty well in agile software development. We have co-located, cross-functional teams that share knowledge. Teams usually struggle on two points.

Firstly, we often don’t have the high level business context i.e. why is our company spending money in this area. This is usually because the company itself doesn’t have a coherent strategy. It’s exacerbated by the fact that (upper) management is not part of the cross functional team and therefore we don’t have direct contact with them.

Secondly, we often don’t have the customer or user context. Sometimes this is because the customer doesn’t exist yet.  Sometimes it’s because we simply don’t have access to them. The most fulfilling projects I have worked on were when we had direct access to the customer.

Principle 2: Have high level goals, not specific ones

A high level goal might be something like “provide customers with an easy way to sign up for electricity connection”. Building software to achieve that goal is already a solution. A very expensive solution.

I have been on a couple of projects where a non-IT solution would have been preferable but we’d already assembled a software development team and couldn’t change tack.

Some companies are spending more time on product validation before the IT build begins. But most companies just start writing software. They move straight into specific goals and lose sight of the bigger picture.

Principle 3: Start small and iterate

We are getting better at this. There’s movement away from monolithic systems and product development is experimenting with ideas around MVP and build, measure, learn.

However, iterating implies starting with a functional first attempt and then re-visiting it to tweak, strengthen and harden it based on feedback. This re-visiting happens very rarely. It is perceived as “re-work” and seen as waste. As a result, there is a stigma attached to it.

Most agile projects have way too many features in the pipeline and not enough time to deliver them. We are too busy churning out new features to properly iterate on the solution.

Principle 4: Go Slow

This whole concept is anathema to most software delivery teams. We have “delivery managers”. We track velocity and cycle time. We use kanban boards to make sure that we are maximising flow. Everybody wants to know how we can go faster, not slower.

This is built into the DNA of corporations. Budgets are drawn up and expenditure is fixed in advance. Each project has a fixed amount of time and money before it even begins. You simply don’t have the luxury of releasing something, taking feedback and then reacting.

Principle 5: Maximise Optionality

Despite some good steps towards reducing complexity such as having “2 pizza teams”, the average software development project is still too complex. As a result, we are constantly bombarded by events that threaten delivery of the project. These include technical considerations but also internal organisational issues.

This results in a defensive mindset. There’s no room for keeping an eye out for opportunities. There’s no time to learn.

Principle 6: Embrace Randomness

Organisations have their own political structures. People have to lobby and compete for resources to get a project underway. In order to do that they have to tell a compelling story.

Randomness changes that story. This causes a political headache for whoever is advocating the story. Good middle managers quickly learn how to spin these changes and adjust the narrative. But this takes time and energy and you only get a couple of chances to change the story. After that, you appear as wishy-washy or incompetent.

Randomness also becomes harder to incorporate the more code you have written. If you change your mind and force people to throw away their work and start again, you burn some goodwill.

As a result of these dynamics, teams are averse to randomness.

Principle 7: Find the right people

In most organisations, teams are put together on an ad hoc basis.  (Note: there are a few organisations that have experimented with allowing employees to self select onto a team.)

As team size is still too big and turnover is quite high due to holidays and people leaving the company, the odds of putting together a genuinely high performing team are very low.

The reason our teams are bigger than they need to be is to hedge against the possibility that two or three people quit the company at the same time and thus threaten the delivery of the project. Companies prioritise stability and predictability over high performance.

In summary, agile software development falls short of being a living design process on the following grounds:-

  • We don’t have a holistic approach. Even when teams are told about the business goals, they are not actively involved in shaping those goals. You are there to deliver software, not to develop a business. The business still thinks of IT as a delivery function. In Jeff Patton’s words, this is the vendor-client anti-pattern.
  • We don’t have the high level goal in mind. By the time a team is formed, somebody else has determined both the business focus and the high level technical implementation. Individual team members are not encouraged to think about these things.
  • We don’t iterate properly. The focus is on getting as many features in before you run out of time and money. This is called a Feature Factory.
  • We often have fixed dates which prevent an open-ended exploration of the problem space.
  • Teams are still too big and are randomly put together.

Many of these issues reinforce each other. For example, because teams are too big there’s more complexity and more expense. Because it’s complex, you are always on the back foot and in a defensive mindset.  Because it’s expensive, you run out of time and money quicker. Because you’re running out of time and money, you can’t take the time to explore the problem space. Therefore, you can’t capitalise on opportunities. Therefore, you can’t pivot properly in the face of new information etc.

Some of these problems are hardwired into the organisation. The corporate governance structure imposes specific limitations on how funding is granted, who is to be held responsible for decisions etc. The most interesting companies in this space such as Gore and Semco have addressed this issue by changing their governance structures.

At one of my earlier jobs, the CIO got up in front of the IT department and told us that “the business would never become agile”. Many years later it’s increasingly obvious to me that this is where the problem lies. We have pushed as much as possible from the bottom up. The final steps towards a “true agile” are now at the organisational level.

Living Design Process Part 2

Following on from Part 1 of this blog, in this post I’ll attempt to extract some of the principles I (think I) followed in my house and garden project in Werribee based on Dan Palmer’s Living Design Process workshop.

Principle 1: Embed Yourself in the Context

I lived in the house in Werribee for six months before undertaking the renovation. I got to know the climate, the prevailing winds, the house’s orientation to the sun, where were the nice places to sit at different times and seasons, which rooms I liked best for certain activities, which parts of the house were in good condition and which weren’t etc.  This understanding was invaluable when it came to making decisions on the renovation such as placement of windows, type of cladding, type of insulation etc.

In the garden, I came to see which areas were sunny, shady, exposed to the wind, how the soil was in different parts of the garden (much more variety than I would have thought), what sort of trees grew in the area (based on observing neighbouring yards) etc. While playing around with vegetable patches, I found the areas of the garden where things grew best. For example, on the west fence there was a space where the previous owner had kept birds in a large birdcage. Anything I planted there grew very strongly which I can only guess is because the soil had years and years of manure to fertilise it. By contrast, there were rocky areas at the back fence where things struggled to grow at all.

The knowledge gained from being in the place allows you to create something truly in tune with its surroundings. This is part of the reason that the cookie cutter housing estates we build these days seem so wrong. They are not in harmony with their environment.  They are centrally planned designs that are inserted into an environment without attunement to local conditions.

Principle 2: Have High Level Goals, Not Specific Ones

This enables you to change your mind based on new information. When I first moved into the house at Werribee, I knew I wanted to get some gardening experience and to experiment with energy efficiency measures on the house. Originally, I thought this meant growing annual vegetables and retrofitting the house. But as I got to know more about the context, including my preferences and desires, the way I achieved these goals changed. I ended up renovating instead of retrofitting and I planted an Edible Forest Garden instead of raised vegetable beds. I still achieved the same high level goals, but in a very different way. The result was something that I was much happier with.

Principle 3: Start Small and Iterate

With the garden, I started with a few raised beds and then gradually added more. As I got sick of watering I explored irrigation options such as Ollas and drip lines. Then I moved to wicking beds. Eventually I transitioned to the edible forest garden idea, but I planted only a few guilds at a time over about a two year period.

I (accidentally) pursued the same strategy with the house improvements. Starting by renovating the front of the house, waiting a year or so and then gradually finishing it off.

Starting small and iterating helps to reduce risk because you’re breaking everything up into small chunks. This spreads the cost out over time and allows you to maximise your learning. You keep the things that work and ditch the things that don’t.

Things that I kept: using IBC totes as water tanks, planting olive and apple trees (which seemed to grow best), cold composting (low labor but longer lead times), building onto existing frames, my builder (he was great), the plasterer, the second plumber I used and planting local native species in the garden.

Things that I ditched after they didn’t work: the first plumber I used (an asshole), solar air heaters (good idea but not practical in this context), raised garden beds (more work than I really wanted), hot composting (a lot of work that isn’t really necessary when you can get the same results from “cold” composting).

Principle 4: Go “slow”

So much of what we do in the modern world seems to have unnecessary haste built in. I see this all the time in my work. Because everything needs to be done yesterday we put together large teams to try and get it done.  That adds massive complexity and risk which is why many projects outright fail to deliver.

For the house renovation, I was in a position to take a slow approach. We had a very small team of myself, the builder and a couple of extra tradesmen. This allowed a very low stress approach the be taken. Low stress is exactly what you want when you’re dealing with complex things like house construction and gardening.  The higher the stress, the more likely you will make a decision you regret.

Going slow allows you time to reflect on what you’ve done. To observe it and understand how it changes things. As you add things to a context you open up new possibilities and close off others. We think we can imagine these possibilities in advance, but the reality is we’ll always miss something. It’s not until you’ve done something that you can fully appreciate what it means, digest it and incorporate it into the next round of decisions.

Principle 5: Maximise Optionality

There is a basic equation which states that the longer you wait to make a decision, the more information you have. Thus, it’s good to put off decisions as long as possible.  When you have the most information, you can make the best choice.

This is known as increasing optionality. As you put certain things into place, new options open up for consideration. From my house renovation, this was best captured by the french door episode. Installing the french doors led to a series of options opening up which would not have been possible if we had simply installed a standard window like I had originally planned.

Similarly, by starting my gardening efforts with only a few raised garden beds I was able to keep my options for the rest of the garden open.  This allowed me to pivot once I knew that the edible forest garden concept was the right one for me.

Principle 6: Embrace Randomness

In our culture we have this idea that randomness is bad. But randomness is only bad because we orientate ourselves to it incorrectly.

Let’s take a house build. Most houses are built via a big upfront planning process.  You have the architectural drawings and layout, all the supporting diagrams, you get together a big schedule of materials and buy them all upfront, you line up the different trades as best you can to minimise downtime. You plan for the most optimal sequence of events.

As a result of this way of working, any randomness, anything that happens that you didn’t expect is a negative.  It throws your plans out. Maybe it rained on the building site and some materials were damaged. Now you have to buy more. Maybe you ordered the wrong size door (then someone like me gets to buy it really cheap). All kinds of things can happen and they inevitably cause delays and put your perfectly ordered sequence of events out of whack. We didn’t get the wiring done in time so now the plasterer can’t come in and we have to push back the plumber etc etc.

But if you embed yourself in the context, go “slow”, start small and iterate, break things up into small chunks and maximise information you position yourself to adapt to change. This means you can benefit from randomness.

When people began to visit the house after the renovation was complete, many complimented me on the french doors. “Good decision” they would say. “Those look great”. But I didn’t choose the french doors. The french doors chose me. If they hadn’t come up for sale, I probably would have bought standard lounge room windows. So, I can’t really take credit for the decision.

What I can take credit for is being open to possibility. When the french doors presented themselves to me, I was able to capitalise because I hadn’t made the decision upfront.  I was free to imagine what they could do for the house. Picking up brand new doors for a fraction of the price is what we would call “good luck”.  What it really is, is being able to turn randomness to your own advantage.

When you have a big upfront plan, you have no “good luck”. You only have “bad luck”. In fact, you’re almost guaranteed to have bad luck. That’s why almost all large projects with big upfront design run over time and budget. We don’t usually like to hold people responsible for these overruns. After all, it’s just “bad luck”. In reality, the process is the problem. Big upfront design causes you to suffer from randomness. Small, iterative, ongoing design allows you to benefit from randomness.

Principle 7: Find the right people

The builder I used for the renovation was recommended to me by a friend.  This gave us a personal connection alongside the business relationship that we were developing. We met several times before starting and I got to know and like his approach. He had done this kind of thing before and I could tell that he enjoyed his job and wasn’t just in it to maximise his revenue.  He was an active partner throughout the renovation and made many important design decisions.

Long and complex projects inevitably have ups and downs and you want people alongside you who can ride it out.  Living Design implies the creation of something that is deeply meaningful.  If all parties aren’t able to find meaning in the project, the project itself will suffer.  You need to have people who find meaning in their work.  You also need to be willing to allow them the freedom and space to contribute their own meaning.  That means giving up some control and putting your faith in others.