Author: Simon Sheridan

Is agile software development a living design process?

I recently wrote a post outlining some principles for doing Living Design Process, a concept developed by permaculturist, Dan Palmer. These were inspired by my experience with a house renovation and permaculture-inspired garden project.

As I was reading back over the post, I realised that these principles were familiar to me from my work in agile software development.

In theory, agile software development should be an example of a living design process. In practice, it’s not. I’ll use the 7 principles from my blog post to explain why.

Principle 1: Embed yourself in the context

We do this pretty well in agile software development. We have co-located, cross-functional teams that share knowledge. Teams usually struggle on two points.

Firstly, we often don’t have the high level business context i.e. why is our company spending money in this area. This is usually because the company itself doesn’t have a coherent strategy. It’s exacerbated by the fact that (upper) management is not part of the cross functional team and therefore we don’t have direct contact with them.

Secondly, we often don’t have the customer or user context. Sometimes this is because the customer doesn’t exist yet.  Sometimes it’s because we simply don’t have access to them. The most fulfilling projects I have worked on were when we had direct access to the customer.

Principle 2: Have high level goals, not specific ones

A high level goal might be something like “provide customers with an easy way to sign up for electricity connection”. Building software to achieve that goal is already a solution. A very expensive solution.

I have been on a couple of projects where a non-IT solution would have been preferable but we’d already assembled a software development team and couldn’t change tack.

Some companies are spending more time on product validation before the IT build begins. But most companies just start writing software. They move straight into specific goals and lose sight of the bigger picture.

Principle 3: Start small and iterate

We are getting better at this. There’s movement away from monolithic systems and product development is experimenting with ideas around MVP and build, measure, learn.

However, iterating implies starting with a functional first attempt and then re-visiting it to tweak, strengthen and harden it based on feedback. This re-visiting happens very rarely. It is perceived as “re-work” and seen as waste. As a result, there is a stigma attached to it.

Most agile projects have way too many features in the pipeline and not enough time to deliver them. We are too busy churning out new features to properly iterate on the solution.

Principle 4: Go Slow

This whole concept is anathema to most software delivery teams. We have “delivery managers”. We track velocity and cycle time. We use kanban boards to make sure that we are maximising flow. Everybody wants to know how we can go faster, not slower.

This is built into the DNA of corporations. Budgets are drawn up and expenditure is fixed in advance. Each project has a fixed amount of time and money before it even begins. You simply don’t have the luxury of releasing something, taking feedback and then reacting.

Principle 5: Maximise Optionality

Despite some good steps towards reducing complexity such as having “2 pizza teams”, the average software development project is still too complex. As a result, we are constantly bombarded by events that threaten delivery of the project. These include technical considerations but also internal organisational issues.

This results in a defensive mindset. There’s no room for keeping an eye out for opportunities. There’s no time to learn.

Principle 6: Embrace Randomness

Organisations have their own political structures. People have to lobby and compete for resources to get a project underway. In order to do that they have to tell a compelling story.

Randomness changes that story. This causes a political headache for whoever is advocating the story. Good middle managers quickly learn how to spin these changes and adjust the narrative. But this takes time and energy and you only get a couple of chances to change the story. After that, you appear as wishy-washy or incompetent.

Randomness also becomes harder to incorporate the more code you have written. If you change your mind and force people to throw away their work and start again, you burn some goodwill.

As a result of these dynamics, teams are averse to randomness.

Principle 7: Find the right people

In most organisations, teams are put together on an ad hoc basis.  (Note: there are a few organisations that have experimented with allowing employees to self select onto a team.)

As team size is still too big and turnover is quite high due to holidays and people leaving the company, the odds of putting together a genuinely high performing team are very low.

The reason our teams are bigger than they need to be is to hedge against the possibility that two or three people quit the company at the same time and thus threaten the delivery of the project. Companies prioritise stability and predictability over high performance.

In summary, agile software development falls short of being a living design process on the following grounds:-

  • We don’t have a holistic approach. Even when teams are told about the business goals, they are not actively involved in shaping those goals. You are there to deliver software, not to develop a business. The business still thinks of IT as a delivery function. In Jeff Patton’s words, this is the vendor-client anti-pattern.
  • We don’t have the high level goal in mind. By the time a team is formed, somebody else has determined both the business focus and the high level technical implementation. Individual team members are not encouraged to think about these things.
  • We don’t iterate properly. The focus is on getting as many features in before you run out of time and money. This is called a Feature Factory.
  • We often have fixed dates which prevent an open-ended exploration of the problem space.
  • Teams are still too big and are randomly put together.

Many of these issues reinforce each other. For example, because teams are too big there’s more complexity and more expense. Because it’s complex, you are always on the back foot and in a defensive mindset.  Because it’s expensive, you run out of time and money quicker. Because you’re running out of time and money, you can’t take the time to explore the problem space. Therefore, you can’t capitalise on opportunities. Therefore, you can’t pivot properly in the face of new information etc.

Some of these problems are hardwired into the organisation. The corporate governance structure imposes specific limitations on how funding is granted, who is to be held responsible for decisions etc. The most interesting companies in this space such as Gore and Semco have addressed this issue by changing their governance structures.

At one of my earlier jobs, the CIO got up in front of the IT department and told us that “the business would never become agile”. Many years later it’s increasingly obvious to me that this is where the problem lies. We have pushed as much as possible from the bottom up. The final steps towards a “true agile” are now at the organisational level.

Living Design Process Part 2

Following on from Part 1 of this blog, in this post I’ll attempt to extract some of the principles I (think I) followed in my house and garden project in Werribee based on Dan Palmer’s Living Design Process workshop.

Principle 1: Embed Yourself in the Context

I lived in the house in Werribee for six months before undertaking the renovation. I got to know the climate, the prevailing winds, the house’s orientation to the sun, where were the nice places to sit at different times and seasons, which rooms I liked best for certain activities, which parts of the house were in good condition and which weren’t etc.  This understanding was invaluable when it came to making decisions on the renovation such as placement of windows, type of cladding, type of insulation etc.

In the garden, I came to see which areas were sunny, shady, exposed to the wind, how the soil was in different parts of the garden (much more variety than I would have thought), what sort of trees grew in the area (based on observing neighbouring yards) etc. While playing around with vegetable patches, I found the areas of the garden where things grew best. For example, on the west fence there was a space where the previous owner had kept birds in a large birdcage. Anything I planted there grew very strongly which I can only guess is because the soil had years and years of manure to fertilise it. By contrast, there were rocky areas at the back fence where things struggled to grow at all.

The knowledge gained from being in the place allows you to create something truly in tune with its surroundings. This is part of the reason that the cookie cutter housing estates we build these days seem so wrong. They are not in harmony with their environment.  They are centrally planned designs that are inserted into an environment without attunement to local conditions.

Principle 2: Have High Level Goals, Not Specific Ones

This enables you to change your mind based on new information. When I first moved into the house at Werribee, I knew I wanted to get some gardening experience and to experiment with energy efficiency measures on the house. Originally, I thought this meant growing annual vegetables and retrofitting the house. But as I got to know more about the context, including my preferences and desires, the way I achieved these goals changed. I ended up renovating instead of retrofitting and I planted an Edible Forest Garden instead of raised vegetable beds. I still achieved the same high level goals, but in a very different way. The result was something that I was much happier with.

Principle 3: Start Small and Iterate

With the garden, I started with a few raised beds and then gradually added more. As I got sick of watering I explored irrigation options such as Ollas and drip lines. Then I moved to wicking beds. Eventually I transitioned to the edible forest garden idea, but I planted only a few guilds at a time over about a two year period.

I (accidentally) pursued the same strategy with the house improvements. Starting by renovating the front of the house, waiting a year or so and then gradually finishing it off.

Starting small and iterating helps to reduce risk because you’re breaking everything up into small chunks. This spreads the cost out over time and allows you to maximise your learning. You keep the things that work and ditch the things that don’t.

Things that I kept: using IBC totes as water tanks, planting olive and apple trees (which seemed to grow best), cold composting (low labor but longer lead times), building onto existing frames, my builder (he was great), the plasterer, the second plumber I used and planting local native species in the garden.

Things that I ditched after they didn’t work: the first plumber I used (an asshole), solar air heaters (good idea but not practical in this context), raised garden beds (more work than I really wanted), hot composting (a lot of work that isn’t really necessary when you can get the same results from “cold” composting).

Principle 4: Go “slow”

So much of what we do in the modern world seems to have unnecessary haste built in. I see this all the time in my work. Because everything needs to be done yesterday we put together large teams to try and get it done.  That adds massive complexity and risk which is why many projects outright fail to deliver.

For the house renovation, I was in a position to take a slow approach. We had a very small team of myself, the builder and a couple of extra tradesmen. This allowed a very low stress approach the be taken. Low stress is exactly what you want when you’re dealing with complex things like house construction and gardening.  The higher the stress, the more likely you will make a decision you regret.

Going slow allows you time to reflect on what you’ve done. To observe it and understand how it changes things. As you add things to a context you open up new possibilities and close off others. We think we can imagine these possibilities in advance, but the reality is we’ll always miss something. It’s not until you’ve done something that you can fully appreciate what it means, digest it and incorporate it into the next round of decisions.

Principle 5: Maximise Optionality

There is a basic equation which states that the longer you wait to make a decision, the more information you have. Thus, it’s good to put off decisions as long as possible.  When you have the most information, you can make the best choice.

This is known as increasing optionality. As you put certain things into place, new options open up for consideration. From my house renovation, this was best captured by the french door episode. Installing the french doors led to a series of options opening up which would not have been possible if we had simply installed a standard window like I had originally planned.

Similarly, by starting my gardening efforts with only a few raised garden beds I was able to keep my options for the rest of the garden open.  This allowed me to pivot once I knew that the edible forest garden concept was the right one for me.

Principle 6: Embrace Randomness

In our culture we have this idea that randomness is bad. But randomness is only bad because we orientate ourselves to it incorrectly.

Let’s take a house build. Most houses are built via a big upfront planning process.  You have the architectural drawings and layout, all the supporting diagrams, you get together a big schedule of materials and buy them all upfront, you line up the different trades as best you can to minimise downtime. You plan for the most optimal sequence of events.

As a result of this way of working, any randomness, anything that happens that you didn’t expect is a negative.  It throws your plans out. Maybe it rained on the building site and some materials were damaged. Now you have to buy more. Maybe you ordered the wrong size door (then someone like me gets to buy it really cheap). All kinds of things can happen and they inevitably cause delays and put your perfectly ordered sequence of events out of whack. We didn’t get the wiring done in time so now the plasterer can’t come in and we have to push back the plumber etc etc.

But if you embed yourself in the context, go “slow”, start small and iterate, break things up into small chunks and maximise information you position yourself to adapt to change. This means you can benefit from randomness.

When people began to visit the house after the renovation was complete, many complimented me on the french doors. “Good decision” they would say. “Those look great”. But I didn’t choose the french doors. The french doors chose me. If they hadn’t come up for sale, I probably would have bought standard lounge room windows. So, I can’t really take credit for the decision.

What I can take credit for is being open to possibility. When the french doors presented themselves to me, I was able to capitalise because I hadn’t made the decision upfront.  I was free to imagine what they could do for the house. Picking up brand new doors for a fraction of the price is what we would call “good luck”.  What it really is, is being able to turn randomness to your own advantage.

When you have a big upfront plan, you have no “good luck”. You only have “bad luck”. In fact, you’re almost guaranteed to have bad luck. That’s why almost all large projects with big upfront design run over time and budget. We don’t usually like to hold people responsible for these overruns. After all, it’s just “bad luck”. In reality, the process is the problem. Big upfront design causes you to suffer from randomness. Small, iterative, ongoing design allows you to benefit from randomness.

Principle 7: Find the right people

The builder I used for the renovation was recommended to me by a friend.  This gave us a personal connection alongside the business relationship that we were developing. We met several times before starting and I got to know and like his approach. He had done this kind of thing before and I could tell that he enjoyed his job and wasn’t just in it to maximise his revenue.  He was an active partner throughout the renovation and made many important design decisions.

Long and complex projects inevitably have ups and downs and you want people alongside you who can ride it out.  Living Design implies the creation of something that is deeply meaningful.  If all parties aren’t able to find meaning in the project, the project itself will suffer.  You need to have people who find meaning in their work.  You also need to be willing to allow them the freedom and space to contribute their own meaning.  That means giving up some control and putting your faith in others.

 

Living Design Process Part 1

Last weekend I attended the Living Design Process workshop run by Dan Palmer. The idea for the workshop arose out of Dan’s dissatisfaction with the way many permaculture projects are run. He has begun to formulate a new approach to both house and garden design that substitutes big upfront planning for an iterative, adaptive and immersive process that he calls Living Design Process.

As the day unfolded it became clear to me that I had unwittingly used a very similar process with my house renovation and garden construction in Werribee. I had found that process to be very fulfilling but had never stopped to think why. Given that Dan has started to put some conceptual structure around that very topic, I want to document the process I went through as I think it’s a nice example of Living Design Process.

For the sake of clarity, I’ll break the post into two parts. The first will give a basic chronological account of what I did in Werribee. In the second, I’ll attempt to relate back to Dan’s Living Design Process concept and draw out some of the principles I think are at work there. Here’s the story.

In February 2014 I bought this house in Werribee.

Not the prettiest thing you’ve ever seen but it had two things going for it 1) it was cheap; 2) it had a decent sized block of land.

I had two high level goals I wanted to achieve at the property. Firstly, I wanted to gain some skills and experience at urban gardening. Secondly, I had become interested in energy efficiency and wanted to see what difference you could make to a house by retrofitting energy efficient materials and technologies. I approached the exercise with an experimental spirit. I knew that I didn’t know much about what I was doing, but I was keen to learn. The fact that the property was cheap and the house old and degraded meant that I had little downside. If things went wrong, I didn’t have much to lose as the property was really only worth the land value anyway.

So, I moved into this old, rundown house with crusty old carpets, dirty walls, chipped paints and all the rest. My first goal was to get the garden going. I kicked off with about 8 raised beds and tried my hand at the usual annual vegetables such as beans, broccoli, carrots etc. I tried raising my own seedlings and killed more than I care to remember. In general, I made all the usual beginner mistakes. I also got to eat food that I had grown which was very satisfying.

The house itself wasn’t exactly a paragon of energy efficiency. Whatever the temperature was outside, that was the temperature inside. By July this meant that overnight temperatures inside the house were down to 3 and 4 degrees. This surprised me as the windows and doors seemed to close properly and there were no obvious drafts. I was to subsequently learn that the devil is in the details – the sealing around the windows, the sealing around the architraves and skirting boards, the insulation (or lack thereof) in the ceilings and walls etc.

I started to wonder what to do with the house. I toyed with the idea of trying some energy efficiency retrofitting such as caulking up holes or adding glad wrap to the windows for cheap double glazing. But the reality was that the house was really old and rundown. It didn’t seem worth the effort.

My thoughts turned to renovation. The main body of the house was quite small and could probably be renovated relatively cheaply. I had a friend who recommended a good builder. He came out and had a look and came up with a ballpark figure which was within the amount I was prepared to pay. By mid-August we were underway.

We had made a time and materials agreement rather than a fixed price quote. To offset some of the cost, I took several weeks off work to be the second pair of hands on the job mainly doing the grunt work that would normally be done by a labourer. Although I didn’t know it at the time, both of these factors would be crucial in getting a great outcome.

As it was just myself and the builder, the pace of construction was fairly relaxed. This also turned out to be a crucial factor. It meant that design could happen as we went. Because everything was stripped back to the frame, we were free to consider a wider range of possibilities as we had the ability to modify the frame where necessary, for example to fit in a larger window. I ended up buying quite a lot of the materials as we went. Among other great finds was a wonderful old kitchen bench that was being thrown out of a very expensive house in Brighton.

A great example of this ongoing design process was the front door. Originally it had been in the middle of the house at the front. I thought it would be better to have it on the side of the house coming straight off the driveway. But that required a bit of extra work to firm up the frame on the side of the house. The builder and I discussed several different options each with their own pros and cons.

After about a week of mulling over it and not knowing which way to go, I came across a brand new pair of french doors that were being sold dirt cheap. They had been ordered as part of a new build but were the wrong size. I checked the measurements against the lounge room wall and realised that they would fit very nicely and would give almost floor to ceiling glass on the north side of the lounge. I ran the idea past the builder and we both agreed it would be a great addition to the lounge room.

This solved the front door problem (it would now go on the side of the house like I wanted it) but more importantly it opened up new possibilities. Now you could walk out of the lounge into the front yard. By coincidence, there was already the makings of a porch there as the old house had a strange kind of room in that space and the roof was still attached. With a bit of extra work we could make a very nice front porch.

So now I had french doors leading to a porch and a porch leading to an outdoor space (the front yard). The front yard itself was already essentially a small courtyard which suited the whole concept perfectly. With one seemingly innocuous opportunity (the french doors) several great design decisions snapped into place and the design of the front of the house and front yard came into being. The lounge room is the best room in the house in winter when the afternoon sun floods in through the french doors, heating up the room and filling it with light.  Even better, the porch blocks the summer sun from the french doors meaning unwanted heat is kept out of the house in the warmer months.

The last major decision of the reno was what to do with the floor. The floorboards were in good condition and I liked the idea of sanding and polishing them up. I tried to get some quotes but the tradies didn’t even want to show up to measure, they just gave a price per square metre figure. Only one guy did come and visit and he was something special. You could tell this guy loved his job. He knew exactly what sort of boards they were (they were actually a rare Tasmanian oak that were a very unusual dimension being several millimetres narrower than the standard). I gave him the job and he really went above and beyond and the floorboards to this day are a wonderful feature of the house.

(The finished product).

With the reno over, summer was fast approaching and that meant getting the raised vegetable beds in order. Although I had set up some basic irrigation (using Ollas), it was still a chore to keep the vegetables watered over a very hot summer. I quickly came to the realisation of exactly how much work was involved in this whole vegetable growing game. I started to hunt around for alternatives. That was when I came across the Edible Forest Garden books. Over summer I eagerly read the books from cover to cover and began to plan out my own forest garden. Here are a few different designs for the same patch to give an idea of the variety of options you can go through.

As I look back on these now I have to chuckle as they bear no resemblance to what I actually planted. I spent a lot of time pouring over these designs that never actually got used. But the exercise of designing was enjoyable. In hindsight, I think the main benefit of the exercise was to slow me down and make me consider things in some detail before taking action.

As soon as the heat of summer was over, I got down to planting the first lot of guilds which you can see here. And you can see the rest of the garden progress in the posts I made over the next couple of years.

During this time I was also finding my preferred composting method after playing around with hot composting, experimenting with solar air heaters, water tanks and greenhouses. These small, inexpensive and low risk experiments turned out to be really useful and the idea of using IBC totes as water tanks worked so well that I ended up using those exclusively in the garden.

By 2016, having enjoyed the results of the renovation to the main house, my thoughts turned to the back of the house. This consisted of an old fashioned outhouse and bathroom plus a small bedroom. Getting up in the middle of the night in winter and having to walk outside to go to the bathroom apparently isn’t most people’s idea of a good time.  So having an enclosed bathroom and toilet seemed like a good way to improve the house.

I toyed with the idea of pulling down the structures and building from scratch but it went against my retrofit idea. The next best thing could be to build walls using the existing framing and turning the back patio area into a living space. Using the same builder and method as last time (go with what works) we did exactly that.

With the house fully renovated and the garden fully planted, I now found myself in a bit of a lull. It was great to be able to walk into the backyard and pick something for dinner but there’s was no more formative work to be done, just the fine tuning.

That would have been fine but there were other tensions at play including the two hours of daily commuting to work, which was getting rather tiresome. The property was now in good shape and it seemed like an appropriate time to cut my ties with it, which is exactly what I did.

So ends the first round of this post. In the second, I will attempt to add some structure to these events and explain why it was that I found this process so rewarding and how it relates to Dan Palmer’s Living Process Design.

(Read Part 2 of this post)