We Are Gonna Need It

There was a time (not so long ago) when we designed everything before we wrote any code.

We gathered the requirements of our application and wrote the requirement specification. We took these requirements and designed an architecture that helped us to fulfill them. We wrote an architecture design document as a guide that we followed when implemented our application.

Unfortunately we never got a chance to do these things.

We were only developers who had to follow plans written by other people. Our job was to write code by following the architecture that was designed by the architects.

Nevertheless, our job should have been simple. We knew all requirements and we had an architecture that should help us to solve any technical problems that we might face.

That was a nice dream. Unfortunately things did not go as planned:

  • The requirements were not correct. To make matters worse, we found this out after we had already implemented most (or all) of them. This meant that we had to rewrite large parts of the application. This took time and money.
  • The architecture did not help us because the architects thought that they are solving a well structured problem that has only one correct solution. Unfortunately for them, software projects solve ill structured problems. This meant that we had no possibility to learn and we had to follow the original plan no matter what. Bad decisions were not overruled because that would have made the architects look bad.
  • We were frustrated because we had no control over our work. We were essentially just typewriters.

Then we heard about agile software development.

Big Design Up Front Is an Anti-Pattern

The Agile Manifesto promised to set us free. It states that:

"Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more."

We got excited. We started to believe that the big design up front is an anti-pattern that should be avoided. We started writing code by following these principles:

We had only one rule:

"Do the simplest thing that could possibly work"

It made perfect sense to us. The upfront architecture design did not make our job easier. In fact, it made our job harder. We were burned by the waterfall model and we wanted to do something totally different. We abandoned the upfront architecture design and decided to solve all technical problems when we ran into them.

This worked pretty well in small software projects (and in the beginning of a bigger software project). However, when we abandoned the upfront architecture design, we started to ignore the consequences of our decisions.

We were excited, motivated, and we thought that we were doing the right thing. But the truth was that our actions caused three very common problems:

  • We wrote our code in small increments and altered it based on the feedback given by our customer. We should have stopped and refactored our code before moving on to our next task, but this requires discipline and we didn’t have it.
  • We didn’t handle corner cases, exceptional situations, or errors properly. We did the simplest thing that could possibly work and because handling these situations was hard, we decided to implement it later when it was really needed. The problem was that when it was needed, our code was already such a mess that fixing it would have taken too long. That is why we decided to simply write an error message to the log file and moved on.
  • We used different patterns, frameworks, or libraries for solving the same problem. Because we had no technical authority, everyone of us chose the “best” tool for the job and used it. We created a code base that suffers from the lava layer anti-pattern. This might have happened during multiple years, but I have seen this happen during the first month of a greenfield project.

It seems that abandoning the upfront architecture design made us happier and probably helped us to add more value to our customer. However, it did not help us to build better software.

We prioritized our short term productivity over our long term productivity. We created a big ball of mud and ensured that maintaining our application is pain in the ass uncomfortable.

"Upfront Design" Done Right

Am I exaggerating? You bet I am, but my experience has shown to me that if the team does no upfront design, they are very likely to make these mistakes. I have seen this happen over and over again, and the odds are that you have seen this too.

That is why I think that we can benefit from the upfront architecture design, but we should not overdo it. We must remember that the goal of the “old fashioned” upfront design is to find the only way to solve the customer’s problem. We don’t need this kind of upfront design.

We need upfront design that does not tie our hands. We need upfront design that helps us to keep our options open as long as possible and doesn’t prevent us from changing things that we don’t get right at the first time.

It is hard to find the balance between the “old fashioned” upfront design and no design, but it is definitely possible. We can get started by following these five rules:

  • We should know our tools. If a certain problem is often solved in the same way, there is probably a good reason for that. That is why we should consider using that method as well.
  • We should avoid best practices that aren’t helping us to do a better job. Common sense is the most important tool of a software developer, and we should always remember to use it.
  • Before we write any code, we should design how we handle the cross cutting concerns such as error handling, validation, transactions, and security. We should also have a rough idea about the architecture of our application, but we should not carve it in stone. It is just a sketch and we will (or we should) update it when we learn more about the customer’s problem.
  • When we implement a single feature, we should do the simplest thing that could possibly work. After we have written the code, we must evaluate the consequences of our decisions. If our changes have a negative effect to the architecture of our application, we must refactor or rewrite our code. This is a favor that we must do to the developers who maintain our application after we have moved on.
  • We must ensure that every member of our team is an architect. If all team members are allowed to participate to the architecture design, they are more likely to follow the common guidelines because they helped us to create them.

P.S. If you want to learn more about this topic, you should read the doctoral dissertation of Veli-Pekka Eloranta.

9 comments… add one
  • Brandon Jan 19, 2015 @ 3:21

    This is smart. I like it.

    • Petri Jan 20, 2015 @ 17:26

      Thank you for your kind words. I really appreciate them.

  • Anonymous Jan 19, 2015 @ 7:27

    Discipline is key, I believe. All team members should have discipline to say to the customer, who is pressing them for more features in a shorter time, that to ensure longevity of the product, refactoring or rewriting of X needs to be done. Not after the next feature, now. But unfortunately my experience has proven that everybody does not have discipline. Some developers are just pure lazy or not competent enough. Combine this with pressure from a customer, and you have a situation where the customer happily accepts the offshore developers saying that "we know there's a need to refactor X, but it will take MASSIVE AMOUNTS OF TIME". As a result, the team moves on to developing the next feature instead of fixing the crap they leave around. The developers can now cover their asses by saying the customer wanted new features instead of fixes to old features. This repeats time after time. The competent developers of the team may attempt fixing things, but there's just too much to fix for them alone. In the long run the customer starts to see the effect of all this. Very odd bugs pop up, repeated attempts are made at fixing the same feature etc. What happens to the development team in this scenario, when the customer finally realises that they lacked discipline and/or competence? They're gradually replaced with more competent developers. And when they start fixing things, we end up with lava flow in the codebase. Pretty inevitable in these scenarios. Also, the original developers, they had their chance of building a future for themselves with the product they were building. But instead they shot themselves in the foot with the lack of discipline. Job well done.

  • John Jan 25, 2015 @ 14:37

    In the agile world also the awareness for design-up-front grows. Please, view a very good explanation of Gojko Adzic, the author of the book "Specification by Example", how teams should use 20% of their time within an iteration / sprint to prepare the next ones:

    • Petri Jan 27, 2015 @ 17:44

      Hi John,

      I have read Gojko's book (Specification by Example) and I liked it a lot. Also, I agree that agile teams should definitely prepare for the next iterations / sprints during the current iteration / sprint. It is sad that so many teams ignore this advice because I have noticed that these teams often face the problems I described in this blog post.

      I will watch his talk and see if I can learn something new from it. Thank you for sharing it!

  • Magnus Jan 27, 2015 @ 1:18

    Very good points. I think you hit the nail on the head with your last point regarding that every team member should be an architect. I've noticed the issues you raise in large corporations where it's more common to have ivory tower architects that isn't embedded with the team that implements it and won't feel the pain points if/when flaws of the architecture is found and raised. If you don't feel the pain points, you're less likely to be concerned about the pain points.

    • Petri Jan 31, 2015 @ 19:52

      It is sad that large companies have these positions mainly because they want to provide a career path for developers that don't want to become managers. In other words, if you want to progress in your career and you work for a large corporation, you either have to move to the ivory tower or get a new job.

Leave a Reply