Introduction
I always feel that I need to be able to logically deduce the logic behind everything I do and agile is no different. At the start of my career, I was never given a proper introduction to agile and it subsequently made me reject the idea until much later when I finally decided to do my own research. This document is a result of my research. I do my best to describe agile in a way that I believe engineers (but not just engineers) will be more receptive to.
If you like what you read, I’d be happy to present this to your teams as well! Just reach out to me on LinkedIn: https://www.linkedin.com/in/pejmanpoh/
Or reach out to me at [email protected]
What will this document attempt to do?
A general observation that I have made is that all engineers are introduced to agile in the same way. They are told that the company is agile, that agile makes better products and all they need to do is attend the rituals. But the problem is that agile is a mindset. And no amount of processes can make someone do a mindset. A mindset needs to be taught and trained. As a result, engineers who don’t understand agile are naturally inclined to reject it as they see it as extra work that doesn’t bring them any value. This document will attempt to logically deduce why agile is the best tool when working in a dynamic environment with customers.
Like any good mathematical theory, it uses “axioms” to establish a basic set of rules that, if understood and agreed with, necessarily lead to the conclusion that agile is the best tool for this environment. This is why this document needs to be challenged, if a single axiom is not true, then the whole methodology falls apart.
Baseline: We are building something for someone
Preamble: Waterfall isn’t inherently bad. It's a linear approach that is useful when you have a clear understanding of the goals with little to no ambiguity. When cooking a dish or building IKEA furniture, Waterfall is the best and simplest approach. However, in dynamic and evolving projects where requirements may change, waterfall isn’t flexible enough to adapt gracefully.
The problem with this is that we are ultimately building these features for users who are not us. They don’t think like us, they don’t work like us, and there’s only so much discovery you can do initially to figure out what they truly need. At the end of the day, to truly know how useful your feature is for them, you need to give it to them and see if they like it. And 9 times out of 10, they won’t. Feedback is crucial in optimising a product to being exactly what the user needs.
Waterfall by its name implies a unidirectional flow of something. If you never get feedback on what you’ve built, you will never be able to improve it. I mean, nobody has told you it isn’t perfect, right? Improvement means redesigning based on the feedback, then developing, testing and re-deploying. An almost… circular way of development.
Without feedback, we are forced to make the first version the final version. There’s not going to be a version 2 so we have to do it all now, and we better hope it’s perfect. This means that projects can take months to build everything from scratch. However, then we run into a problem. How long… exactly?
Axiom 1: It’s very hard to estimate well
Raise your hand if you’ve been burned by this before: A project that was supposed to take 2 months ended up taking 3 months, stakeholders unhappy, team is stressed, hacks and shortcuts are taken, code becomes worse. We say to ourselves “Next time, we’re really going to get it right”, and then next time comes around and we realise that we didn’t cut enough and some things we cut too much! By the second sprint, the whole project has been restructured, reprioritised, and reorganised.
Seems familiar? The logical trap we fall into is that we believe that the more we estimate, the better we become at it, improving in a linear fashion (red line below). Eventually, if we estimate enough, we’re going to get so good at it our margin of error is on the scale of seconds! However, the truth is that we improve in a logarithmic fashion (blue line below).
At some point, we stop getting better at estimating. There are just too many variables to consider and our brains hit a limit on how much it can take into account. There are also simply unknown variables that won’t be known at the start of a project, not even including other unexpected diversions e.g. meetings, incidents, etc... There is a natural margin of error that stops estimates from ever getting better past a certain point.
If I tell you that it took 10 minutes to draw the cat on the left. Can you logically deduce exactly how long it’s going to take to draw the dog on the right?
There are similarities for sure. It’s a four-legged animal with a tail. But there are also differences. The dog has floppy ears, shorter and stumpier legs, and a different facial structure. Chances are the cat artist is initially going to accidentally make the dog look a little too much like a cat because they were so used to drawing cats and they will need to erase parts of it to try and make it look more dog-like. They will need to iterate.
The issue with product development is that we are never drawing the same animal twice. Every feature we release is a brand new animal with a brand new set of differences. Which means we can’t reliably say that, “it took us 10 minutes to draw the cat, so it’ll probably take us around 10 minutes to draw the dog.” Even if the dog only took 20 minutes, that’s still a 100% increase in time required. And no stakeholder is going to be happy with a 100% increase in time.
Conclusion: It is virtually impossible to estimate correctly.
Axiom 2: We still need to estimate
I just spent ages trying to convince you that we can’t estimate. Then this axiom says we should estimate. How does that make any sense?
Unfortunately in the real world, telling our clients and stakeholders “it’ll be ready when it’s ready” does not build a healthy relationship. Stakeholders naturally need to know some kind of timeline in order to do their own work. So we should come up with a compromise.
Axiom 1 states that “it’s very hard to estimate well”. But how hard definitely depends on how large the task is. If I ask a dev to change a translation key, the dev should be able to tell me down to the minute how long it will take. Meanwhile, asking a dev how long it will take to implement an AI-powered blockchain NL Processor in CSS may as well be rolling a dice. The trend here is the larger the feature, the larger the margin of error. So Axiom 2 should really be “It’s very hard to estimate larger features well”.
The commonly used Fibonacci sequence for estimating tasks replicates this exponential growth in uncertainty well. If we switch the numbers to days, we might get something like this:
If we assume a blanket 50% margin of error for all estimations, then you’d be seeing ± 1.5 days for 3 day estimates and ± 1 week for 2 week estimates etc…
This means when we want to find out what kind of estimates we should give our stakeholders, we need to find a compromise where they are comfortable with the margin of error.
Axiom 3: Promising X weeks of work is a great compromise
In company A, any feature being late by 1.5 months would be unacceptable. For company A, the maximum a feature can be late by is at most 1 week. That means we are only allowed to do 2 week estimations max (because 2 week estimations have a margin of error of ± 1 week).
Company B however may be different, this is entirely dependent on the industry. Hence why the axiom doesn’t specify the value and says “X” instead. For the sake of this document, let’s assume we’re working with company A and 2 week sprints.
Now, let’s go back to our hypothetical 3 month project at the start of Axiom 1. With a 3 month estimate we are looking at a margin of error of ± 1.5 month. Which we established is too large of a margin of error for company A.
So let’s cut down the 3 month feature into 2 week chunks!
Great, now instead of promising a feature in 3 months, we promise 1/6th of a feature every 2 weeks. We are now agile!
This is where most company’s stop. Most companies think that if they work in sprints, they are automatically agile but like was established in the beginning, just doing processes doesn’t make you agile. If you gave the above image to an engineer, their response would be:
What the hell difference does this make? It’s still going to take me 3 months to build the damn thing.
This is from my experience the general attitude that engineers have against agile. And they would be right! If this is as far as you go with agile, then it doesn’t actually make any sense. You’re just doing waterfall with extra steps.
The key difference with this system is the word promise.
We are promising 1/6th of a feature in 2 weeks. This means we need something deliverable in 2 weeks.
Now, we have a goal behind our feature cutting. We need to give our stakeholders something they can actually use in 2 weeks. That means cutting basically everything that isn’t 100% essential. This means cutting things all the way down to the colour. Make the feature black and white and raw HTML if it needs to be.
Then the beauty is the user can dictate what the next iteration should look like. They can tell us what the current iteration is missing or if there’s a bug. They can tell us if the UI doesn’t make sense or if the data structure doesn’t work for their use case. And then we take that feedback and let it inform the next iteration, and the next. And whether you like it or not, you will be forced to build what the user actually wants!
And even if a feature is late by a week, that’s ok. We established with our stakeholders that a 1 week delay is still acceptable.
Now you may be thinking: “Won’t stakeholders be equally upset that the feature looks terrible and is so barebones?”
This is the bet that agile is making. That a stakeholder who can physically see and touch a prototype, no matter how shitty, will not get mad about how long something is taking. Because they can physically touch the progress.
There we have a mindset switch. The stakeholder’s expectations stop becoming “when is the project finally going to finish and be released?” to “how long until the feature is good enough?”
As long as the feature continues to improve based on the feedback of the users at a reasonable pace, stakeholders will be happy.
With this expectation, we open up a series of exciting opportunities.
1. We get the user’s feedback early
This point is deliberately number 1 because it is by far the most valuable. Instead of building something for 3 months only to figure out it is not useful to the user at all, they can tell us that right from the beginning.
A user telling us something isn’t user friendly early saves us a ton of headache later on when that feature is more established. As long as we do this correctly, we will be able to build almost exactly what the user wants/needs.
2. We no longer need to worry about estimating correctly from the beginning
With the waterfall way of working, a 3 month estimate becomes a 3 month deadline as soon as it is communicated to the first stakeholder. Then, any change in scope, any unknown variable becomes a new source of stress on the deadline. This goes against the reality of product development where new information is constantly being revealed.
By working with agile iterations, the stakeholders are much less likely to be upset when you miss your 3 months estimate because they already had the product in their hands by week 2. It was barebones but it was there and every 2 weeks it improved. By the 3 month mark, it may not be where it needs to be but stakeholders will be a lot more understanding that a bit more work is required and will be willing to be patient if we’ve spent 3 months building up trust with them by constantly asking and implementing their feedback.
3. We have more clarity on the product
In a waterfall 3 month project, the goal of sprint 1 is generally: “Build some of it… like… about 1/6th of it.”
With agile, we have a very strong sprint 1 goal: “Build this exact MVP version with x, y, and z requirements.”
This is a lot more motivating and a lot clearer what work needs to be done. This not only gives clarity to our stakeholders to know exactly what to expect every iteration, but this clarity also enables team members to be a lot more autonomous because everyone knows where the project needs to be by the end of the two weeks.
It also acts as a useful early warning system on features. Missing a sprint goal is, in itself, not a big deal. It happens. But repeated misses for the same goal is a good red flag that something is going wrong.
Sounds good, how do we get there?
Agile is a mindset. And with any mindset, understanding and embracing a mindset is more than following a set of processes; it's about internalising a particular way of thinking. By ingraining this perspective, when faced with a question on how to tackle a task, one should be able to draw upon these fundamental principles to arrive at the right solution.
Just implementing a bunch of agile processes while still thinking in terms of waterfall will result in an awkward situation where you do processes that don’t seem to provide any value. For example, Scrum rituals that are seemingly done just for the sake of doing them.
It will take time to transition into the agile mindset. It will be a muscle that you will need to consciously train. Every time a situation comes up about how we should do something, we should sit down and think: “What is the agile way of solving this problem?”
In essence, we need to change our systems, our processes, and most importantly our mindset.
Systems: We need infrastructure that allows us to test quicker. We need the wider company to understand this new more iterative approach otherwise it will just look like we are trying to release ugly half-baked features. We need users we can test prototypes with, not just internally with CS.
Processes: We need to better solidify how to turn stories into tests and then iterate again. We need to figure out how best to communicate with each other and collaborate (e.g. stand-ups and retros).
Mindset: We need to adapt our own mindset. It’s really easy to accidentally fall back into the waterfall mindset.
Conclusion
Agile is a mindset. And like a muscle, a mindset needs to be trained. Doing agile means thinking about feedback loops and integrating feedback as early as possible into every process you have in order to build the best product. To do this, one must be prepared to make compromises by estimating something that is inherently very difficult to estimate. But the compromise will be worth it as you will unintentionally build the best product for your user. But achieving this is a journey of iterations itself. You must test out what works and what doesn’t for your situation. And even if you end up with something not at all like I described above, I am certain all that reflection has led you to a better place regardless.
Images sourced from Vecteezy.com