“We will have this project wrapped up next month”. That was the 3rd month running it would be done next month. And since in technology the cost of delay is mainly people cost, we were now close to 20% over budget. This wasn’t a low key project either. It was a multi year investment moving internal operations to a new platform.
A classic tale of a late project getting later.
Sounds familiar?
The Challenge of IT Project Cost Estimation
I know each late project is late in its own way, to paraphrase Tolstoy. But really, something can only be late because we predicted an end-date. So when estimate IT projects should we try and predict dates at all? Or is it all just management theater.
This classic tale of a late project getting later raises important questions about how to estimate IT projects and the necessity of planning when estimating project timelines.
When Planning a Project, It Is Necessary to Estimate: A Desire for Predictability
Let’s be clear: predictability has its place. There are plenty of scenarios where meeting a delivery date matters. For instance in my time at The Sun, having our London Olympics 2012 section ready had a very real deadline. The Olympics weren’t going to delay their start because we weren’t ready. Our readers certainly would switch to other sources. And as a “free” news business, no readers means no advertising revenue. It doesn’t have to be this dramatic of course. It could simply be to agree on a kick off date with the marketing team. Which in turn drives their schedule.
The biggest enemy of predictability is uncertainty. And uncertainty in software has 2 big sources. Requirements driven uncertainty, and technical uncertainty. Let me unpack.
Requirements driven uncertainty is obvious to most people. It is near impossible to perfectly capture the entire requirement to a level of detail that creates full certainty. How often did you do “a small tweak” along the way? Or discussed a “refinement” or added a “quick clarification”? Those are all good things by the way. It means you’re likely reacting to new information. The world isn’t static after all. But each of those have an impact on your initial prediction. And during the life of developing software, these micro decisions on requirements get made daily by product managers and engineers. They uncover these along the way, but didn’t know about them when doing the initial prediction. And so you’ve introduced uncertainty in the estimate because your requirement wasn’t 100% clear.
The second part of uncertainty is on the technology side. In technology we stand on the shoulders of midgets, not giants. No Atlas to carry us. Instead, we’re building on a patchwork of code. Some by previous developers in the company, some open source code, some more established frameworks, and some random bits copied from the internet. That holds up remarkably well for the most part. And experienced developers often have a good sense of the limitations of that Jenga tower. But extending an existing codebase is an inherent source of uncertainty. There are ways to manage this, but you can’t eliminate it.
The hardest scenario is where technical and requirements uncertainty meet. That often means introducing something brand new in the product. Let’s say we wanted our software to paint a room. This means we’d have to define what a room is in our system. We’d also have to introduce paint and its properties such as color and drying time. And define at least one way to apply paint to a wall. You see where this goes. The good news? Once you’ve solved this for 1 room, there is almost no added uncertainty for 10 rooms. Often in software, the complexity is in new features rather than repetition.
Uncertainty is the biggest distraction from predictability. I always advocate to talk about uncertainty and predictability at the same time. So rather than “when”, the question is “When, and how likely?”. Or in a similar line of thinking, a window of time. It’ll be done in more than 6 but less than 12 months. More uncertainty means a wider window.
IT Project Estimation Methods: Using Throughput as a Key Indicator To Counter Changes in Uncertainty
If you agree that uncertainty is a key element in predictability, it stands to reason we want to find ways of observing changes in it. One approach could certainly be an index built by daily survey of all developers’s level of uncertainty about the prediction; that might not be practical. Instead I often recommend looking at throughput.
Throughput is a measure of how much work a team gets done over time. It is of course tempting to look at every single task and judge it against some ideal throughput. I have found that counter productive.. In reality, some tasks will go faster and some will take a long time. That’s the nature of dealing with the inherent complexity of technology.
Instead, put a few weeks of a team’s work together. It will account for some positive and negative surprises. And a stable, mature, team tends towards a steady throughput in my experience. Don’t let this measure become a goal or a target though. Use it as an indicator. Changes in throughput, up or down, are highly correlated with changes in uncertainty.
The real value for you as a leader: changes in uncertainty should prompt you to ask questions! Back to my story from the intro. It turns out the team was working in a very complex codebase, caused by years of neglect. If we had looked at throughput of that team we’d have noticed a slow and steady decline once the initial few truly easy changes were done. Finding that out after 2 months, rather than 18, would have been nice. And it would have allowed us much more options than putting the team on a death march. Bonus points for morale, bonus points for budget, bonus points for realistic expectations. And when I say realistic expectations, I mean expectations based on facts.
Conclusion
So to bring this all together. Yes, estimation matters. And delivering on estimates matters too. But estimates need to expand to include a level of uncertainty. Make it a range, or a likelihood of delivering by a point in time.
There is an economic tradeoff between reducing that uncertainty up front and the level of accuracy of the estimate. You can investigate more, do technical research etc. In theory, you could create the entire product. That would give you all the knowledge needed to accurately predict how long it takes. Not practical of course. So the key is to strike a balance between a level of uncertainty you can work with. And then an ongoing way of tracking your level of certainty.
That’s where high level throughput is your key gauge. Changes in that should prompt you to dive in, and take action.
By being explicit about uncertainty, you are focusing the conversation on understanding sources of uncertainty. Not “can we hit this date”, but rather, what are the key areas that impact our ability to deliver.
