Hopefully I caught your attention with the controversial (and admittedly hyperbolic) title. Good. Let me restate it in a slightly more elegant and less clickbaity form:
In general, you can either have software delivered on time or good software, but you canât have both*
* except on counted occasions in established, highly-performant teams
The past few months Iâve been reflecting about why delivering good software does not play well with estimates and planning in general. During my career Iâve seen projects ran in many different ways (waterfall, true agile, agile-fall) and all of them shared a thing in common: no matter what kind of project we were working on, if we âdid it rightâ (i.e. we didnât take hacky shortcuts that would give us nightmares), we always ran past the deadline.
On the other hand, whenever a project was delivered âon timeâ, either its scope was reduced along the way or we had to cut so many corners that the tech debt accumulated during implementation all but guaranteed that the project would have to be re-written shortly after launch, which made me wonder: Is a project actually âon timeâ even when the end result is an uglier, less maintainable, bug-ridden and, plainly speaking, suckier version of what we set out to do initially?
Iâve also worked on projects that were not deadline-driven. There were âdeadlinesâ, sure, but far from them being set in stone, it was understood that deadlines were flexible and that quality trumped on-time delivery. Those were the projects which produced the best software, had the happiest developers and were the most successful overall amongst the ones Iâve worked on. But we all know those projects are a rare find, or I wouldnât be writing this.
So why is it so difficult to plan and deliver good software given a static timeline? I think it has a lot to do with creativity, craftsmanship and unpredictability.
The creative side of coding
I am of the view that software development is an inherently creative undertaking. Sure, there are developers who perform repetitive, trivial tasks, but they will have a job only as long as no one figures out how to automate theirsââânot a great place to be in, and not the focus of this essay.
For me, thereâs something special about the act of creating something new and looking for original solutions to challenges that calls me to software development, and I donât think I am the only one. In fact, I believe creativity is the main reason why developers enjoy their work. In my experience, whenever Iâve worked in environments where there were strict and unchangeable âset of practicesâ (think tech stack, processes, guidelines, etc.)âââi.e. the least creative freedom I hadâââ, the less engaged I felt. âAfter all, if they already have it all figured out, why do they need me?â, I wondered. On the other hand, I felt much more fulfilled, excited and productive at work when there were few top-down directives, I was given creative leeway and was trusted to own up to my technical decisions.
Itâs important to note that more creative freedom begets more trial and error to reach the solutionâââwhich is fine. Some people think that you can just know the perfect solution ex-ante (i.e. in advance) before writing a single line of code. I contend instead that, for creative activities, the process of discovering a solution to a given problem (not only software) is a process of tinkering: you canât have perfect knowledge in advance, but instead you learn by doing, iteratively trying new things and keeping what works, and by refining your solution (and possibly shipping it to your customers if you are lean/agile) until you are satisfied with it.
Just think of how many times you have spent time designing a feature on paper, just to have to change the design completely once you actually start implementing it. There will always be unknown unknowns and the only way to uncover and deal with them is ex-post by actually coding your solution, instead of spending a lot of time theorizing and pretending we have perfect information in advance. This process of tinkering does not play very well with estimates.
Further, as itâs the case with other creative endeavors, coding benefits from Strategic ProcrastinationâââA term coined by Adam Grant which asserts that often you canât be creative on demand but instead creativity comes as a âpush eventâ from a process running in the background of your mind:
â Employees who procrastinated regularly spent more time engaging in divergent thinking and were rated as significantly more creative by their supervisors. Procrastination didnât always fuel creativity: if the employees werenât intrinsically motivated to solve a major problem, stalling just set them behind. But when they were passionate about coming up with new ideas, putting off the task led them to more creative solutions.â
â Grant, Adam. âOriginals: How Non-Conformists Move the World.â
Again, not great news for the central planners out there trying to plan and measure every minute of software development projects.
The craft of building software
The best developers I know are crafts[wo]men. Craftsmanship is the hallmark of good software: you donât only build something that works, but you build it in the best way possibleâââItâs relatively easy to build something that works, but very hard to build something that works and stands the test of time.
As a crafts[wo]man, the quality of your work defines you. You prime quality over quantity because you donât want to be known to write crappy software, even if you could satisfy your Product Manager by just making your software look good âon the outsideâ but full of nasty surprises on the insideâââwhat I call Kinder Surprise Development. You know that investing time in writing good software pays off and you push back on pressures to âexecute fasterâ as you know the more shortcuts you take now, the less life expectancy your code will have and the more problems it will cause down the road.
My eyes bleed
Craftsmanship is about caring: caring to do a great job, caring about the ones who will maintain your code after you, caring about the consumers of your software having an easy time using it, caring about your team mates, and so on. You care because you are not an asshole and because you know that itâs the right thing to do if you want the project to be successful.
In short, good engineers have the difficult task of caring about qualityâââsomething that has the long-term consequencesâââin a world driven by short-term execution.
In practice, that translates into things like:
- Finding the right mix between encapsulation, extensibility, scalability, etc.âââagain, you will need iterative trial and error, no one will build the best solution on the first try.
- Taking time to refactor when you stumble upon a shamefully bad part of the code
- Writing good, complete testsâââmaybe even doing TDD
- Code pairing with colleagues
Needless to say, itâs impossible to plan for all these in advance, so this doesnât help you hit any deadlines either.
Your predictions are wrong
Even with clear requirementsâââand it seems that they never areâââit is still almost impossible to know how long something will take, because weâve never done it before. If we had done it before, weâd just give it to you.
â Ron Jeffries, The NoEstimates Movement
Software Projects are Complex Systems: They are made possible by human beings and thus are affected by interpersonal interactions, motivation, communication issues and human psychology in generalâââall those pretty difficult to model and quantify on a spreadsheet, if you ask me. This makes software projects very difficult to model (and hence predict). Nassim Taleb explains it best in his book Antifragile:
âComplex systems are full of interdependenciesâââhard to detectâââand nonlinear responses. âNonlinearâ means that when you double the dose of, say, a medication, or when you double the number of employees in a factory, you donât get twice the initial effect, but rather a lot more or a lot less. Two weekends in Philadelphia are not twice as pleasant as a single oneâââIâve tried.â
â Nassim Nicholas Taleb, Antifragile
Whatâs worse, given that time canât be negative, any unplanned âsurprisesâ will very likely increase the completion time instead of reducing it as there is an asymmetry in the outcomes:
âJust as time cannot be negative, a three-month project cannot be completed in zero or negative time. So, on a timeline going left to right, errors add to the right end, not the left end of it. If uncertainty were linear we would observe some projects completed extremely early (just as we would arrive sometimes very early, sometimes very late). But this is not the case.â
â Nassim Nicholas Taleb, Antifragile
This is bad news, as uncertainty is certain and even small errors in estimating individual tasks will compound exponentially at the project level. All this is assuming the best case scenario where deadlines have been set after some careful estimation by the developers, but reality is even more absurd: most of the time âThe Businessâ sets the deadlines arbitrarily and only then Engineering comes up with a plan to deliver the requirements by those randomly chosen points in time, in a flagrant case of start building the house from the roof and then putting the cart before the horse on top.
Good question
Here are some examples to illustrate some of the nonlinearities and feedback loops of writing software:
- That time when you assumed that the API you need to communicate with accepts
accountIdbut it actually only acceptsmemberId. You had to add 4 days to your estimate to refactor the API codeâââwhich in turn had to go through a separate review process which added 2 additional days. - A task estimated to 2 days ends up taking a week because during the review process one of your team mates pushes you (and rightly so) to refactor and improve a very ugly part of the code that was long overdue.
- That one-point task where you had to implement some new functionality that happened to require an update to a dependency which ended up taking 4 days as the dependency update triggered a chain reaction of dependencies needing to be updated and that caused a bunch of errors in the build.
Are we screwed?
We keep playing the estimation and planning game out of inertia to reassure ourselves that we know what we are doing, but the truth is that we donât, and that software projects are empirically unpredictable. Thus, in my opinion, weâd be better off just focusing more on doing rather than planningâââ#NoEstimates anyone? But of course this will not fly in many organizations: âWe canât just let the engineers run the show, unchecked, there needs to be some accountability!â. I get it.
You donât say?
What to do then? I think it comes down to bridging the gap between the world of the spreadsheet and the world of the IDE in a way that allows maximum creativity, flexibility and craftsmanship to Engineering while at the same time carefully managing the commitments made to, and the expectations from the project stakeholders. The Engineering Manager is in the best position to bridge and manage this gap and act like a buffer between the two worlds. Itâs not an easy job, but a necessary one. Aaron Longwell explains it well in his article:
âBecause engineering managers inhabit the border between the business and the technical, they are the ones who need to resolve the tension between estimate and reality. Itâs a bit like being the wishbone being pulled at both ends; either side can snap. When the business side âwinsâ, the developers end up in a death march. When development concerns outweigh business ones, you end up blowing the budget and deadline. Either way youâre broken. Successful software managers find ways to be flexible_; to bend without breaking and to resolve the tension gradually. Servant leadership can be a guide to finding this flexibility.â_
â Aaron Longwell, Why Software Development Requires Servant Leaders
Itâs also very important to have a strong relationship based on trust between Product and Engineering. If there is trust, you will be able to negotiate timelines honestly and openly with good faith. If you have previously proven that your team delivers good software, there should be enough âsocial capitalâ for the stakeholders to trust that if you push back on a given schedule is for good reasons and for the common good.
Another âtrickâ that Iâve personally used as a Manager is to avoid giving concrete dates, as they will inevitably be used as hard deadlines. Diffuse time ranges work best, something like âbetween 3 and 5 weeksâ. Then, the closer the diffuse deadline gets, the more definition you add to it: âBetween April and Mayâ turns to âBetween April 15th and May 3rdâ at the beginning of April, which turns to âThe week of April 20thâ around April 10th, and so on. This way you are openly truthful with your peers while allowing your team the flexibility they will need to solve the inevitable unforeseen problems that will arise.
Lastly, remember that itâs up to the developers to stand up for the technical quality of the product delivered, not to the other stakeholders. Itâs only natural that tensions will arise between organizations with seemingly divergent incentives on the surface. The key is to point out that, in fact, you are all (presumably) aligned with the same goal: Deliver good quality software to the customers in the fastest way possibleâââthe gist that only good developers seem to understand is that you should avoid taking the deceivingly âfast and easyâ way as itâs actually the slowest one in the long run.
In conclusion, itâs a solvable yet complex (and common) problem. Iâd say that if you feel like your manager or your organization doesnât set you up to build good software, itâs up to you to speak up and try to change it or, if that fails, find somewhere else to work.