The Psychology Behind Late-Stage Software Testing

Suppose you had the choice of receiving 100€ today or 110€ in a year. If you are impatient like me, you would probably choose the 100€ today. But how would your decision change if you had to choose between 100€ today and 100 000€ in a year?

And what if we turned the decision around and gave you the choice of having to pay either 100€ today or 100 000€ in a year? Sounds easy? Interestingly, in software testing, many people choose the more expensive option in the future.

A bug that would cost 100€ to fix in the early stages of software development can end up costing 100 000€ if it has to be fixed in production (see rule of ten below). Thus, in theory, implementing early testing should be a no-brainer. Yet, we see many development teams that fail to do so. Behavioral psychology suggests that this tendency can be traced back to the way our brain perceives losses and gains at different points in time.

The Problem With Late-Stage Software Testing

Software bugs are inevitable. While there is some debate as to which combinations of testing approaches and tools are most effective at reducing them, most people agree that finding bugs earlier is better than finding them late. For this reason, tech leaders, such as Google and Microsoft, have been “shifting left” for years, and many software teams are following their lead.

According to the rule of ten, the costs of eliminating a bug increase by a factor of ten with each phase of the software development process it passes through undetected. These costs also contain the amount of developer time that is required to fix bugs that were committed months or years ago (often by other developers) and to fix consequential errors.

The reasons that cause some development teams to test late, despite the obvious benefits of doing the opposite, are typically quite rational. For example, when the necessary staffing or tooling for early-stage or continuous software testing isn’t available. But in addition to these rational motives, there are also irrational factors such as biases and cognitive errors that can subconsciously influence decision-makers to opt for late-stage testing.​​

Temporal Discounting in Software Engineering

In its essence, temporal discounting describes our inclination to discount rewards and losses that are in the future. The further they are away, the stronger we discount their value. Therefore, when presented with two choices of equal value, one in the now and one in the distant future, we are likely to choose the one in the now.

Temporal discounting is not just limited to monetary gains and losses. In a 2019 study by Fagerholm et al., software engineers were split into several groups. The groups were told that they were either working on a hypothetical 1-year, 2-year, 3-year, 4-year, 5-year or 10-year project. Next, all participants were presented with two options:

1. Implementing a new feature from the product backlog (5 person days)

2. Integrating a new library (5 person days), that would speed up the development process by N-days. Over the course of the whole project.

Next, participants were asked:

How many days (N) would the new library have to save you throughout the entire project, for you to prefer option two over option one?

The results showed that the longer the overall scope of the project, the more “person-day savings” it took for software engineers to prefer the long-term option. Thus, software engineers valued time-savings less if they were further away in the future. They discounted their value.

Source: Fagerholem et al., 2019

Temporal Discounting in Software Testing

Software testing is no stranger to cognitive biases, as we know of confirmation bias (also known as positive test bias) and inattentional blindness, just to name a few. When it comes to temporal discounting however, software testing has to be treated slightly differently from what we described above, since we are talking about discounted losses, instead of gains. Nonetheless, the principle remains the same: Although we rationally know that the costs (monetary and non-monetary) of unfixed vulnerabilities will increase the longer we delay software testing, it can be hard to grasp the full scope of these consequences, as they are diluted by the factor of time. Thus, from the perspective of today, the higher costs of bugs being discovered late seem a lot smaller than they actually are. Since the effect of temporal discounting depends on the amount of time that passes until we feel the effects of our losses, it is particularly strong in large projects that have a scope of several months or years.

To be fair, most development teams do not make up their software testing strategy out of thin air. Measures such as security audits and data-driven strategy all serve the purpose of making well-informed, rational decisions. But since decision-makers are still human, biases have to be taken into account and residual risk remains. Especially with constrained resources and tight schedules, decision-makers are sometimes forced to make compromises and to act fast. Even when there is enough time, resources, and staffing available to thoroughly plan software testing measures, temporal discounting should be kept in mind, and measures should be taken to minimize its effect.

Preventing Human Error in Software Testing

Although it might sound trivial, being aware of our biases already helps us to make better decisions, as it allows us to take a step back to think about them consciously. This activates the rational part of the brain, which is less susceptible to cognitive errors.

In software testing specifically, an effective approach to reducing cognitive errors is by implementing commitment devices. A commitment device is no more than a conscious decision that binds oneself to a strategic plan and takes away the option to delay. In software testing, opting for automated testing solutions can be such a commitment device. Once decision makers have decided to implement such a tool, developers can automatically run security tests at each pull request, throughout the entire development process. Automated security testing should come easier to decision-makers than manual early-stage testing methods, as it does not slow down the development process. In fact, automated testing solutions even speed up the whole process by eliminating bugs when they are still easy to fix.

Commit to Automated Software Testing

If you are interested in what such a testing approach might look like, you can check out the free demo app of the fully automated application security testing platform CI Fuzz. By directly integrating with any build environment, CI Fuzz empowers developers to run security tests without extensive software security expertise. In the free demo app, you can access real bug findings, code examples, and code coverage reports. You can also contact the dev team to onboard your own projects for continuous security testing. All you need to access the demo app is a working GitHub account.

Sign in with Github

What Do Software Testing and Marshmallows Have in Common?
During the famous Stanford marshmallow experiment in 1972, children were placed in a small room with a marshmallow in front of them. The children were then told that if they managed not to eat the marshmallow for 15 minutes, they would be rewarded with an additional marshmallow. Many children preferred to eat the marshmallow in front of them immediately instead of waiting for the second one. Just like the children who preferred the immediate gratification of eating a marshmallow now, over the discounted gratification of eating 2 marshmallows in 15 minutes, software testers can also prefer the gratification of unimpeded software development now, over the benefits of early testing, which only manifest themselves in the distant future.