Published Friday, 24. September 2010, 01:32, Danish time.

Agile processes are here to stay. But the implementation is hard in many organizations. And the reason for that is that software is a very complex phenomenon. 

In short, producing perfect software is simply impossible given the huge number of varying and unknown parts.

Your software has to match functional needs, communicate clearly to users so there is no misunderstandings, perform well, scale well, be simple to operate, be easy to maintain, be bug free, look and feel attractive and more.

And high quality software is crazy difficult to produce: understanding what you actually need is hard, communication between devs and customers is hard, devs don’t always have the skills needed, you learn and get new ideas along the way and your customer’s market is most probably a moving target.

On top of that, management usually presses to – or is pressed to – make the financial future predictable, so they naturally want to freeze the project expenses. And plan use of all the developers in the next hot project that has to start in exactly 2 months. And lock down the features by selling the product in advance to important clients. And …

And when the project is done, your world changes, your competitor makes an unexpected move, the company reorganizes or new technologies get available. And suddenly your perfect software is outdated and needs to change.

There are also several clashes between traditional thinking and the kind of thinking that’s needed in order to be successful in software projects.

In my mind, the most difficult part is to give up on the idea of certainty. In most software projects, certainty is simply not an option. In my experience the most common conclusion when you look at projects in hindsight, is that they didn’t at all evolve and turn out the way you expected.

So what can you do?

You have to act rational and make sure that every step you take forward makes sense. And take great care along the way. If you are really good, you will probably achieve 50% of what you thought you would. If you are really good, 50% of what you achieve will be good things that you never even thought of in the first place.

I think that “making sure” along the way is the key to success. In fact, I think it’s your only chance!

For my part, I learned about “making sure” the hard way.

Getting washed over by waterfalls

Back when I started developing in the beginning of the 90’es, all projects at my job were waterfall projects. We spent an enormous amount on hours analyzing, planning and arranging Gantt charts. It was hell: not only was it very hard to get right under time pressure – you also often had a bad feeling about it, because you were often forced to make brave assumptions. And every detail had to be adjusted several times. People were often pretty stressed out about it and that usually lead to mistakes.

The downsides of the waterfall approach are well known today:

  • It only works, if you have complete domain knowledge and know exactly what has to be done in advance. And this is (almost) never the case.
  • It doesn’t make room for unexpected problems.
  • It doesn’t give you many chances to learn and correct things along the way.
  • “The business” (the customer) almost never know their business to a degree where all rules were so well known that a developer could express them in code.
  • Using a kind of circular thinking was widespread: if your plan failed, then it was because it wasn’t detailed enough. Then apply more waterfall thinking!
  • When the plan was detailed enough, you had more or less done 75% of the total work in the analysis phase. You simply had to – in order to be able to guarantee a successful result. This became known as prototyping.
  • Cutting scope was often done by the project manager – who often didn’t know the business that well.
  • (I am sure that 50 more items could easily be added to this list…)

It was very frustrating if you had ambitions to make your users happy and deliver well functioning solutions. So grand failures were common. Ironically, the bigger the projects were, the more impossible the whole thing was and the more spectacular failures this method would lead to.

I was very fortunate to start my developer career doing projects where I had a very good domain knowledge myself. I started programming out of need to solve specific problems (and perhaps also because of geeky genes). But even though I thought that I knew my domain stuff well, plans often had to be turned upside down or extended anyway.

It became obvious that there was only one way ahead:

Feedback loops, feedback loops, feedback loops!

You had to communicate. I started talking to colleagues and users every time something was unclear. And every time the plan for a complicated feature was about to be finished and locked down in the project plan I talked to my users to verify the planned solution. Just to make sure.

I would go and ask questions like: “have you considered the consequences of feature X; because of the way it works, you have to carry out task Y in this new way and you can no longer carry out task Z like you used to – is that acceptable?” Or: “if we register your actions during your work, then we can automate the statistics and you don’t have to fill out forms afterwards – but it will mean that your work performance can be measured in detail – what do you think about that?”

There is one particular experience, that I often wonder about: it was often best to hook up with the users that were considered “noisy” by their managers. That was a paradox that took me a while to understand. As I see it now, they were often noisy because of ambitions and pride, topped with frustrations over the state of things. I would go and ask: “what would be the best thing to do?” A simple question with a surprising effect. A fruitful discussion would often unfold. They would be relieved that somebody was listening and I would know a lot more afterwards. Often they became enthusiastic about the project because they could see an end to their frustration. And in several occasions they would become super users and good assets for their employers once the new system was delivered. An amazing paradox!

Back on track: I often came back from lunch or the coffee machine with new and disturbing knowledge and had to re-plan a lot. And re-do Gantt charts and all. Pain! On top of that, this was usually not what made project managers very happy. Getting knowledge about bad design or potential failures when the promises are already made is not nice for anybody… So I had my share of fights…

I had already seen to many failing projects and work process re-designs, so there was really no turning back and I got stubborn about it. Fortunately I was lucky enough to get into projects where the right kind of “close-coupling” with the users were appreciated – because it led to better results.

I don’t think that my story is unique. Fortunately others had a far more generalizing approach and formulated and compiled well thought-out practices like Scrum. But to this day, I still think of Scrum and other agile practices as means to make sure that you are doing the right thing.

Let me elaborate a bit on this.

Scrum is concentrated making-sure philosophy

Here are a couple of examples:

  1. The ScrumMaster role. The ScrumMaster makes sure that developers can work efficiently without interruptions, have a fully functional computer, and so on.
  2. The Daily Scrum meeting. Here the team makes sure that they get progress,  that no dev is thrashing, that unforeseen problems are dealt with in an efficient manner, and so forth. Constant feedback captures mistakes early.
  3. The Product Backlog. This is used to collect and store work tasks, so that the Product Owner can make sure that nothing is forgotten.
  4. The Product Owner must describe and prioritize product backlog tasks in order to make sure that the team always have meaningful work tasks to fill into their sprints.
  5. The Product Owner must also make sure that all the work done in the project makes sense from a business perspective.

And so forth.

In my mind it gets really easy to understand Scrum, if you view it from this angle.

Today it’s clear that agile thinking has won in most of the software business. But there’s still a lot of people who’s not convinced, or bend the Scrum practices to a degree, where you can hardly call their practices for Scrum anymore. Scrum “developer” Ken Schwaber calls these practices for ScrumButs and even wrote a book on the subject. He found that the re-occurring excuse pattern was “Yes, we use Scrum, BUT … we don’t do X, because of Y explanation”. He has also said that “only 25% of all organizations that embraced Scrum would fully benefit” and I have heard his partner in crime Jeff Sutherland talk about a third. Not good rates. A pity.

The general thinking is that ScrumButs flourishes because it’s hard to give up old habits and old culture.

My point is that you don’t really understand software and software project challenges well enough, if you don’t “make sure”. I don’t think you really have a choice.

---

Here is a couple of additional, interesting links I came across while writing this post: