Randomness in Software Estimates

Estimates can be useful when building software. There is the immediate comfort in having expectations, but there are also practical benefits. Estimates allow for the proper prioritization of work. We don’t just work on things that are important, we work on things that give us a good amount of value for the time we have to put into them.

Photo by Mike Szczepanski

Unfortunately, estimates are often inaccurate. The larger the estimate, the larger the margin of error. A 5 minute task will rarely balloon into 5 days, but a 5 month project can easily get out of control. I was once in a meeting discussing a 4 month project that involved 3 teams of developers and how we could get estimates accurate to the day. I pointed out that in the room there was over 100 years of software development experience. Had anyone in the room seen a project that large have an estimate that was accurate to the day? No one responded.

Some believe that if developers just try hard enough or develop their skills in estimation well enough, they can get accurate estimates. However, the margin of error on a large project is going to be unavoidable. There is a certain amount of randomness that will be unaccounted for in project estimation.

To understand this randomness, let’s look at rolling a pair of dice. You may think that a single roll of dice is random and can’t be predicted. That’s not entirely true. Anyone can perfectly predict the next roll of a pair of dice. All you need are:

• The weight of the die
• The amount of force used to throw the dice
• The starting angle and height of the dice
• The amount of air resistance
• The angle of impact with the surface
• The amount of friction on that surface

Is getting all this data realistic? Maybe with enough sensors and a robot throwing the dice. Is it worth trying to do? No. Yet, since gathering this data requires an unreasonable amount of effort, a single roll of dice is seemingly random.

Whether we like it or not, estimating software projects perfectly also requires an unreasonable amount of effort. We can probably get estimates accurate for tasks that will take a single developer a few hours or a few days, but the estimate on a multi-month project with many developers will have so many factors that it will add a seemingly random amount of inaccuracy.

A large part of this is because there are more than just technical factors in building software, but most development teams only account for the technical factors. Software isn’t built by software yet. It is built by people. People have all sorts of things that slow them down.

The most obvious one is fatigue. Being tired increases the likelihood of developers introducing bugs and increases the amount of time needed to fix those bugs. I knew an entire team of developers that spent a long weekend fixing a single bug that ended up being an issue with case sensitivity. These developers weren’t bad developers. They had just been on a death march that had lasted months.

Being tired also affects creativity which is essential in designing software. I remember crunching for a month to get a release out. The last feature took me two 14 hour days. I managed to get some sleep afterwards, but came back the next morning finding all sorts of bugs with it. After looking at the code I had written with fresh eyes, I realized it was inherently flawed. I had chosen a foolish approach that required a lot of code and was never really going to work. I ended up deleting all my work and rewriting a working version in 2 hours. Sleep deprivation made a 2 hour task take 28 and produced an inferior product to boot.

While developer exhaustion can be controlled by good management policies, being physically tired isn’t the only form of fatigue. We’re thinking and feeling human beings. We could be getting plenty of sleep and still feel tired emotionally. Poor mental health can just as easily affect the alertness required in preventing bugs and the creativity required in good software design.

While managers can make sure developers aren’t forced to stay in the office for an unreasonable amount of time, they can’t control every other factor in a developer’s life. Are they having issues with their personal relationships? Are they concerned about the direction of their life? Do they have addictions they need to overcome? Whether we like it or not, whether the developer’s themselves want to prevent it, all these issues bleed into office life and affect productivity.

There are also conflicts that can occur between members of a team. The larger the number of people on a project, the larger the possibility that a few people will simply not get along. People who don’t like each other are less likely to compromise. They’re less likely to make sure the other party is informed. And they are less likely to have productive debates.

Software design is extremely subjective and every developer will have their own preferred approach. Most well functioning teams can have a well reasoned debate based on the merits of each approach. People who don’t like each other are going to resolve that debate with stubbornness. The meetings will take longer and the solution will be poorer, both of which will increase the amount of time taken on a project.

Most managers are going to be unaware of these conflicts. It’s not exactly professional to tell your boss you don’t like someone. But just because it’s not professional doesn’t mean a human isn’t going to have that feeling. It’ll just be bottled up and manifest itself in the work in an unpredictable way that will add inaccuracy to a project estimate.

There is an argument to be made that you can account for all these hidden factors by aggregating estimates and then applying a multiplier like +50% or multiply the estimate by 3. A single event is unpredictable, but aggregation of all events can make an estimate accurate. My main counter to this point is that the same argument was made with mortgage backed securities during the financial crisis. It didn’t quite work there and it doesn’t quite work in estimating software development.

Despite all the randomness that affects the amount of time a project takes, creating estimates is still a worthwhile endeavor. We just need to be aware that estimates are estimates, not exactimates. The estimates are likely to be wrong the larger a project is. The question is not how can we make those estimates more accurate, but how do we handle things when it becomes apparent that they’re not.