Is agile software development a living design process?

I recently wrote a post outlining some principles for doing Living Process Design, a concept developed by permaculturist, Dan Palmer. These were inspired by my experience with a house renovation and permaculture-inspired garden project.

As I was reading back over the post, I realised that these principles were familiar to me from my work in agile software development.

In theory, agile software development should be an example of a living design process. In practice, it’s not. I’ll use the 7 principles from my blog post to explain why.

Principle 1: Embed yourself in the context

We do this pretty well in agile software development. We have co-located, cross-functional teams that share knowledge. Teams usually struggle on two points.

Firstly, we often don’t have the high level business context i.e. why is our company spending money in this area. This is usually because the company itself doesn’t have a coherent strategy. It’s exacerbated by the fact that (upper) management is not part of the cross functional team and therefore we don’t have direct contact with them.

Secondly, we often don’t have the customer or user context. Sometimes this is because the customer doesn’t exist yet.  Sometimes it’s because we simply don’t have access to them. The most fulfilling projects I have worked on were when we had direct access to the customer.

Principle 2: Have high level goals, not specific ones

A high level goal might be something like “provide customers with an easy way to sign up for electricity connection”. Building software to achieve that goal is already a solution. A very expensive solution.

I have been on a couple of projects where a non-IT solution would have been preferable but we’d already assembled a software development team and couldn’t change tack.

Some companies are spending more time on product validation before the IT build begins. But most companies just start writing software. They move straight into specific goals and lose sight of the bigger picture.

Principle 3: Start small and iterate

We are getting better at this. There’s movement away from monolithic systems and product development is experimenting with ideas around MVP and build, measure, learn.

However, iterating implies starting with a functional first attempt and then re-visiting it to tweak, strengthen and harden it based on feedback. This re-visiting happens very rarely. It is perceived as “re-work” and seen as waste. As a result, there is a stigma attached to it.

Most agile projects have way too many features in the pipeline and not enough time to deliver them. We are too busy churning out new features to properly iterate on the solution.

Principle 4: Go Slow

This whole concept is anathema to most software delivery teams. We have “delivery managers”. We track velocity and cycle time. We use kanban boards to make sure that we are maximising flow. Everybody wants to know how we can go faster, not slower.

This is built into the DNA of corporations. Budgets are drawn up and expenditure is fixed in advance. Each project has a fixed amount of time and money before it even begins. You simply don’t have the luxury of releasing something, taking feedback and then reacting.

Principle 5: Maximise Optionality

Despite some good steps towards reducing complexity such as having “2 pizza teams”, the average software development project is still too complex. As a result, we are constantly bombarded by events that threaten delivery of the project. These include technical considerations but also internal organisational issues.

This results in a defensive mindset. There’s no room for keeping an eye out for opportunities. There’s no time to learn.

Principle 6: Embrace Randomness

Organisations have their own political structures. People have to lobby and compete for resources to get a project underway. In order to do that they have to tell a compelling story.

Randomness changes that story. This causes a political headache for whoever is advocating the story. Good middle managers quickly learn how to spin these changes and adjust the narrative. But this takes time and energy and you only get a couple of chances to change the story. After that, you appear as wishy-washy or incompetent.

Randomness also becomes harder to incorporate the more code you have written. If you change your mind and force people to throw away their work and start again, you burn some goodwill.

As a result of these dynamics, teams are averse to randomness.

Principle 7: Find the right people

In most organisations, teams are put together on an ad hoc basis.  (Note: there are a few organisations that have experimented with allowing employees to self select onto a team.)

As team size is still too big and turnover is quite high due to holidays and people leaving the company, the odds of putting together a genuinely high performing team are very low.

The reason our teams are bigger than they need to be is to hedge against the possibility that two or three people quit the company at the same time and thus threaten the delivery of the project. Companies prioritise stability and predictability over high performance.

In summary, agile software development falls short of being a living design process on the following grounds:-

  • We don’t have a holistic approach. Even when teams are told about the business goals, they are not actively involved in shaping those goals. You are there to deliver software, not to develop a business. The business still thinks of IT as a delivery function. In Jeff Patton’s words, this is the vendor-client anti-pattern.
  • We don’t have the high level goal in mind. By the time a team is formed, somebody else has determined both the business focus and the high level technical implementation. Individual team members are not encouraged to think about these things.
  • We don’t iterate properly. The focus is on getting as many features in before you run out of time and money. This called a Feature Factory.
  • We often have fixed dates which prevent an open-ended exploration of the problem space.
  • Teams are still too big and are randomly put together.

Many of these issues reinforce each. For example, because teams are too big there’s more complexity and more expense. Because it’s complex, you are always on the back foot and in a defensive mindset.  Because it’s expensive, you run out of time and money quicker. Because you’re running out of time and money, you can’t take the time to explore the problem space. Therefore, you can’t capitalise on opportunities. Therefore, you can’t pivot properly in the face of new information etc.

Some of these problems are hardwired into the organisation. The corporate governance structure imposes specific limitations on how funding is granted, who is to be held responsible for decisions etc. The most interesting companies in this space such as Gore and Semco have addressed this issue by changing their governance structures.

At one of my earlier jobs, the CIO got up in front of the IT department and told us that “the business would never become agile”. Many years later it’s increasingly obvious to me that this is where the problem lies. We have pushed as much as possible from the bottom up. The final steps towards a “true agile” are now at the organisational level.