Opinion: Estimating Software Development ProjectsBy John Parkinson | Posted 07-20-2006
Opinion: Estimating Software Development Projects
One of the hardest things to do in any human endeavor is to predict the future accurately. Since the future hasn't happened yet, it is, by definition, to some degree uncertain. We may harbor highly developed expectations about what's going to happen, but in most circumstances, we cannot be entirely sure. Yet for many "economic" endeavors, it's important to provide just such a prediction. It's unlikely we will get permission to do something that requires an investment of time, money and other assets if we can't offer some degree of certainty about the outcome.
Software projects are no different. We need to be able to predict how long the work will take, how much it will cost, how much resources will be required to complete the work, and what skills and experience the people involved in the project will need. One school of thought holds that developing software is more akin to developing a new product than to manufacturing something that is well understood. Requirements evolve as features are developed. Some parts of the job turn out to be harder than expected, others easier. Some people work faster than others or are more skilled, so that even at the same pace their work gets completed faster because they see better ways to solve problems. For managers trying to assess the amount of effort and time will be involved, there are many variables to consider and much ambiguity. What should they do to get the best possible answer? If the future is so uncertain, how can we predict anything useful?
Fortunately, we have some things going for us. First, not everything in the future is actually completely uncertain. The real world exhibits statistically predictable phenomenology at a useful level of detail for many events macro (the sun will rise tomorrow) to micro (my alarm will ring at the time it's set for).
Second, we can use history as our guide: We can measure and record what actually happened in the past and (a) identify things that always seem to happen and (b) what things seem to cause differences in outcome (the determinants of variability). Then we can use these observations to build predictive models that can be refined and improved through use, and the more we use them the better we should get.
And third, there are constraints (some based on the laws of the universe, some arbitrary, but just as "real") in the real world that let us eliminate or ignore part of the range of what's theoretically possible as an outcome.
Estimation is the process by which predictions are constructed, tested and offered up with a known degree of confidence. A lot of different estimation methods have been developed (for software and other kinds of development efforts ranging from civil engineering to event planning). But all successful methods fall into a small number of categories, according to the strategies they use to reduce uncertainty. There are three basic approaches.
Algorithmic Estimation (or model-based approaches).
This method works best when past performance has proven to be a good indicator of future outcomes. We "decompose" the problem into a set of structural elements (use cases, classes, methods, database tables, etc.) that can be counted and for each of which we have a reliable estimate (or narrow range of estimates) of how much effort it will take to deliver the result. We then run the numbers through our models, which can account for a range of other factors, such as pre-existing assets (things we can use without needing to build anew); team size; levels of skills and experience; availability of expert inputs and so on. We can even factor in "soft" variables, such as the historic propensity of the customer to change his mind or the probability that a new technology will be delivered late or will underperform expectations.
The more complex the model the more you will need technology to help you use it efficiently, but along with the use of technology comes possibilities like sensitivity analysis (what impact will changing assumptions have?), which helps you identify risks that should be managed closely. In the extreme, you can actually use advanced computational methods (such as genetic algorithms) to "breed" the best estimates from your inputs and assumptions.
Algorithmic estimation seems to have fallen out of favor recently. Some people in the agile software development movement dismiss the approach as irrelevant because there are so many unquantifiable unknowns at the start of a project. But it still has its supporters, including me. A decade ago, I built a sophisticated estimation model (and a tool that made it easy to use) based on an iterative solution to the Traveling Salesman problem that was used to estimate thousands of projects with very high accuracy (I still have the model if anyone wants to see it, although the tool is now obsolete). But the approach does have limitations in circumstances where there isn't enough history to develop a reliable model. In this situation we get to try our next method.
Compare and Contrast
Compare and Contrast
Estimation by Analogy.
Here we use principles of similarity to create our estimates. We may have no data about the current problem, but we do have data about other problems we have solved, some of which will be reasonably similar to the problem at hand. By comparing aspects of the new problem with similar aspects of other problems, we can place our new problem within a reasonable range of things we have already done, and thus derive an estimate of the time and resources it will require.
Humans do this kind of estimating all the time, often unconsciously. "Taller than x," "further than y," "faster than z" are all examples of estimation by analogy, and this familiarity makes the approach both intuitive and popular. It is, however, subject both to systematic bias (people often estimate some things badly using comparative methods, especially when the comparison pushes the limits of their experience) and to outlier bias (when the available experience does not actually represent the actual distribution of possible outcomes very well), and these biases are often hard to detect.
I used to counter these bias issues by asking several different people to make the estimation independently. It was remarkable (a) how different their estimates would be and (b) how well the range of estimates encompassed what actually happened when the project was reviewed.
But what do we do when we have no data and no comparable experience? That's when our third approach comes in.
Heuristic estimation (sometimes called structured guessing).
This method attempts to use abstract reasoning to derive an estimate even when we have no good analogies and no actual experience. Heuristics are "rules of thumb," useful guidelines that have no deterministic proof of applicability but that seem to work. The approach actually has a statistical theory (Bayesian Estimation) behind it, but you don't need to be an applied statistician to use the approach. Just gather a moderately sized group of smart people together and pose the question to them. Then let them debate the answer for a while and see what happens. You can use techniques like estimation poker (where participants "gamble" on the accuracy of their estimates) or Delphi groups (where each participant "solves" the problem independently, all the solutions are reviewed and ranked by the group, and the process is repeated until a consensus emerges). I've used this method with groups as small as 4 and as large as 600, and my observation is that you need about a dozen inputs to avoid the same kinds of bias that analogy exhibits, but you don't really get "better" answers as the group gets larger than about 20. I always tried to mix "experts" and "novices" into the group so I could control for false analogies or the use of inappropriate algorithms by some participants (participants can use any method they want to develop an estimate, including ones that can't possibly work).
Heuristic estimation seems like voodoo to many people, but it's remarkably good at determining upper and lower bounds for estimates, considering how hard the underlying problem actually is.
It should be clear that no single approach to estimation works best in all circumstances. As a result, it's best to use as many approaches as possible to guard against "corner cases": unlikely but possible estimates that can be generated by slightly incorrect assumptions or counts that can often bias a specific approach.
None of the above eliminates the fact that an "estimate" is still not a certain prediction. Perhaps the hardest thing to get across to the customers for estimates (often the "funding authority" for a project) is the dynamic nature of the process we are estimating. Ordinarily, we get around this by offering a range for the estimate (called a "confidence interval" by statisticians). Unfortunately, a principle of human psychology, called the "principle of the excluded mean" by behavioral psychologists, often intervenes. This principle tells us that the estimate range we provide will probably be forgotten by the audience, or rather will be translated mentally so that the lower bound is remembered as the upper bound and the upper bound is forgotten. Not a good outcome, as there is a lot of evidence that software engineers and their managers generate more optimistic than pessimistic estimates compared to the outcomes they achieve.
Next time I'll look at how we translate estimates into plans and what can go wrong with that part of the process.