What are the most effective Agile project estimation techniques?
This is a fantastic question! Estimation is an interesting process in that it is both complicated and straightforward and the quality of estimates varies from one person to another. Software project estimation practices were refined when the waterfall process was dominant, and most of the science and math behind estimation was developed from that perspective. Agile processes are different than waterfall processes, and Agile teams in large organizations frequently work to support waterfall business models. Still, agile project estimation techniques should be slightly different than waterfall methods were.
There's an incongruity there that makes it challenging to maximize the value of incremental development, rapid customer feedback and the ability to pivot. While it can be straightforward to improve the accuracy of estimates, the real goal is to improve the effectiveness of estimates. While more accuracy always helps, it may not be the best way to become more effective.
There's currently a "no estimation" movement, where Agile teams are exploring not estimating anything at all. I believe this is the result of recognizing that an Agile team supporting a waterfall business process is realizing the cost of estimation in time, effort and psychic costs, but may not be realizing the benefits. When the team is doing the "next most important thing" at all times, it doesn't matter to the team how much time it is going to take -- because that doesn't impact how quickly they can complete the work. Peter Saddington wrote a nice summary on the Agile Scout blog.
The point that the detractors of estimation are missing is that estimates are intended primarily to provide value not to the team, but to the rest of the business. The interesting question to the business is not actually how long it will take, but how worthwhile will it be. From a product owner evaluating a single story to a business determining if it should invest in a product at all, the issue is really one of "bang for the buck." The business needs to invest efficiently. That doesn't mean doing the most-valuable things, or the lowest-cost things; it means investing in the things that have the highest return (value) relative to investment (cost). Asking a business to invest in something without prediction about the costs is irresponsible.
While more accuracy always helps, it may not be the best way to become more effective.
From an Agile project point of view, effective Agile project estimation techniques are really about accelerating the rate at which predictions of what will be accomplished are improved. At the time that a project starts, there is a lot of uncertainty about what will ultimately be completed. Individual tasks will take more or less time than estimated. Considering the full scope of a release, the vast majority of the capabilities to be built are undefined -- or at least are subject to change – and, therefore, the individual tasks are not definable. This makes the accuracy of estimates at the start of the project almost irrelevant to the decision of whether or not to fund the product based on a prediction of costs.
Before a project starts, many of an Agile team's task-estimates are irrelevant, because they won't be building everything they just estimated. The uncertainty captured in an early prediction reflects not only errors about how long something will take, but errors reflecting whether that something is created or whether it is replaced with a different something altogether. The further into the future a prediction is, the greater the uncertainty of that prediction.
To improve the effectiveness of estimation, a team needs to provide regular feedback about the continuously improving accuracy of the predictions about what will be delivered, its value and how much it will cost. Most of the waterfall projects I've seen predict a total cost, scope and time to deliver. When the delivery date approaches, the team and the business share a collective sense of surprise as they discover that the team is (way) off track. The best waterfall teams I've seen will do a "20% review," where, after 20% of the time originally estimated has past, they revisit the estimates, to see if the project investment is "still a good idea."
Agile project estimation techniques are made more effective by embracing the idea of incremental, recurring updates to the business. Let stakeholders know what is now intended, what we now think it will cost and when we now think it will be completed. Over time, the error (or confidence interval) in the predictions tightens -- becoming more accurate. Regular updates to the business enable them to provide regular course-correction or affirmation and to increase investment when appropriate. The important thing for a team to focus on with regards to their project estimates is not making the discrete estimates more accurate. Rather, the team should leverage the power of an incremental-delivery process to provide regular predictions to the business, which will enable the business process to be agile too.
This was first published in June 2013