In software development, speed has become the defining characteristic of product launches. It sets the standard we can’t really escape anymore. It places demands on market players both old and new: release now, fix the issues later. Launch the feature today or watch your competitors outpace you. Meet the deadline or lose a vitally important partner/client.
These are all thoughts and ideas most product managers, CTOs, and team leads have had (or heard). In part, I think the culprit for this state of affairs has been automation itself, as it has streamlined so many tech processes across industries and has set client and market expectations.
For software products, however, automation can also be the saving grace, offering a unique opportunity to go both fast and steady. According to Boehm’s law, the cost of spotting and fixing issues grows exponentially with time. This is where automation testing comes into play. Organizations all over the world are automating tests to ensure both velocity and product stability.
The idea is: why choose between quality and speed when automated testing can ensure quality at high speed? I know that’s quite the mouthful, but it’s essentially true.
Now, many companies, product managers, CTOs, and CEOs are hesitantly standing on the doorstep of that ‘automation ride’ and wondering: should I go in? Is it worth it? Will I spend more than I get back? The truth is that automated testing isn’t that different from any other business investment. It all comes down to calculating the returns — here’s how I do it.
The basics of calculating ROI in automation testing
It’s worth noting that in some parts of the automation testing community, the ROI of test automation is considered to be somewhat of a myth. This is obviously a controversial opinion that does not necessarily reflect the overall state of the industry. Still, it may help understand why some automation QAs are skeptical about the idea of ROI in this case.
To delve deeper into the subject of ROI of test automation, let’s start by looking at the basic ROI formula for automation testing. The most widely used way to calculate ROI for automated testing is the following:
ROI for testing automation = Cost savings / Investment cost
OR
(Investment value – Investment cost) / Investment cost
However, each company will have to take into account their own specifics, pre-existing problems as well as natural trends in software development to apply this calculation correctly.
To begin with, cost savings can be calculated in various different ways, based on your knowledge of company processes, expenditures, and the testing automation tools in question.
For example, the go-to method of counting savings is to consider how much Quality Assurance time is saved by automation. If implementing automated testing saves us Z hours of QA each month, it’s easy enough to calculate the X cost of those work hours versus the Y cost of implementing automated testing. Most people see it as a simple matter of inserting X and Y into the first variant of the equation.
If we assume that the cost of implementing automated testing is 500 units and we save 100 units each month on lessened QA time, we’ll break even in 5 months. Beyond that point, we have a constantly increasing positive automated testing ROI.
Next, the investment category of the equation can be vastly different in each specific implementation case. Some companies opt to develop their own automation framework, while others prefer to find external partners that specialize in this area or have ready-made tools.
Needless to say, depending on whether your team has experience in developing and maintaining automated testing frameworks, the costs (and cost-effectiveness) can vary wildly.
Now that we have mostly covered the subject of how to calculate ROI for test automation, let’s talk about another “phantom” point to factor into your analysis: What happens if you don’t invest.
Opportunity costs of manual testing
No discussion about any sort of ROI analysis in automation testing can really be complete without considering the potential financial or organizational losses you might suffer if you don’t invest in new tech. “The cost of lost opportunity”, as it were.
To correctly evaluate the opportunity (i.e. investing in a streamlined process), you have to consider the downsides of the commonplace alternatives. Here’s a brief rundown of the possible lost opportunity costs of sticking with manual testing:
A slow-down of multiple company processes
The average speed of automated testing is about five times that of its manual counterpart. This impacts your delivery velocity, deadlines, team dynamics, stress, and employee burn-out.
That’s only half the issue though. Any in-house department influences other departments. The company is a living, breathing organism.
A slowdown in development or production can also grind other departments to a halt: sales, marketing, auditing, and others. It can place a strain on HR to manage employee happiness or to recruit new developers/testers that couldn’t handle the crunch.
Organizational micromanagement
As with anything that’s done in-house and by humans, there’s always a need for organizational management, adjustments, meetings, briefings, and employee one-on-ones.
If your managers don’t have to deal with that yet, it’s only a question of time. As soon as you scale up, you’ll have to get involved in managing the human side of your teams under the immense pressure of delivery velocity.
Manual testing increases this micromanagement strain since you’re offloading even more technological complexity onto living people that will require support to stay productive.
Lowering of delivery output due to regression tests
As Blake Norrish accurately points out in his blog on “the Regression Death Spiral,” testing isn’t only about verifying that one new feature. There’s also regression testing.
Consider what happens when you’ve been working for years on a software product. Dozens of engineers and developers leave, and new ones arrive. It’s a multi-layered cake of code created by different people.
Each manual test is a laborious process of making sure the new feature doesn’t break any or ALL of the previous ones, created over the ENTIRE period of development.
Ignoring this idea can often lead to disastrous consequences (like a new patch that breaks the fundamental functions of the entire product).
What happens when your team has to do continuous regression testing of each new feature? That’s right — everything slows down painfully. Burn-out, stress, and missed deadlines ensue. Automated testing is the widely accepted solution to avoid both slowing down your testing AND avoid new features breaking old systems.
Overall, only the stakeholders within the company itself can determine whether you need to implement test automation at any given time. However, none of us can run from innovation forever and not ultimately fall behind.
Common mistakes in ROI calculation
As with any new endeavor, making the calculated decision to adopt new tech is only half the journey. How you approach the implementation process and whether you’re wise enough not to repeat the mistakes of others is likely to determine your ROI value of automation vs. manual testing as well.
Thankfully, the industry has made enough strides in the direction of automation for most of the common pitfalls to be glaringly obvious. Let’s take a look:
Ignoring ALL manual testing scenarios
Automated testing doesn’t fit every single scenario and process. A lot of the value of automated testing stems from running these tests multiple times and not every scenario requires automation.
If your development is diverse in its processes and products/features, you may have to run manual tests alongside automated tests. Being discerning where to apply one or the other will save you money.
Not taking into account the training time
Whether you hire an outside team to handle testing automation or launch a project using only the resources you already have, your QA automation ROI calculations should always include time and money spent on training.
This can be the training of manual QA engineers who start working with automation or the training of Automation QAs to work with specific tools and technologies. Moreover, when you intend to outsource your automation testing project to an offshore team, extra time and money may be needed to create a perfect synergy.
Only short-term or narrow ROI calculations
While it’s completely fine to first consider the initial cost and return of implementing test automation, it’s also necessary to evaluate the long-term ROI for your company.
I’ve already outlined the potential problems with manual in-house testing. No doubt you have your own insights into these (and others) since you know your team well.
Will automated testing solve more issues than just QA time in the long run? Will it open up new opportunities? Will it improve the quality of development and product launches? These are all things to consider in long-term ROI calculations.
Ignoring test maintenance
Automated tests are still code. They’re still processes that need to be maintained and updated if you want them to provide value for years. This is a factor you need to investigate in your ROI equation.
Is your team ready to handle the maintenance? Or do you have a partner/outsourcing firm that’s proficient in this? How do you wish to handle maintenance in the long-term?
Ignoring setting up the automation testing environment
To calculate ROI accurately, it is vital to consider the efforts invested into building a testing environment.
“This point does not just concern the process of setting different operating systems, browsers, and mobile devices where tests will be run. It also includes setting up the CI process and tools, such as Docker, or a server such as Jenkins, TeamCity, and so on. We also need to account for time spent installing the necessary tools — for example, for security and load testing. ”
Maxim Khymii, Automation QA Lead, TestFort
Ignoring records/documentation
This is where you address the ‘knowledge leakage’ parameter in automated testing ROI calculations. It’s a short and vital directive:
Document everything you can.
There are few events as damaging as losing a key employee that kept all the knowledge in their head and having to re-engineer numerous complex cases. Automation can help here as well, as there is less of a dependence on human expertise (though it is still obviously necessary).
Depending on your product, organizational structure, and management style, the above considerations may move up or down in priority. You may even encounter entirely new pitfalls as the technological environment changes. Yet keeping these points in mind will help you stay flexible and vigilant.
3 things that can make ROI calculations less accurate
When everything is done right, the automation testing ROI can give you a pretty solid idea of the impact of test automation on your business. However, it’s not always possible to give an absolutely precise calculation because there are some aspects that are hard to account for.
There are always undetectable issues
Achieving 100% test coverage, whether with the use of manual or automated testing, is not achievable at the moment. Therefore, even the most qualified QA team will not be able to spot 100% of issues, and you’ll need to include troubleshooting costs in the overall quality assurance budget. Of course, it’s hard to predict how much locating and resolving those issues will cost in the long run, which makes calculating ROI slightly more difficult.
You cannot survive on automated testing alone
Automation testing and manual testing are not mutually exclusive concepts, as we’ve concluded in one of our recent longreads. This is why, if your QA project still heavily relies on manual testing — as it should — you won’t be able to precisely calculate how much money you’ve saved on automating your tests because manual testing will still play an important role in the progress of the project.
Not every benefit has a monetary value
Some metrics of testing automation, such as the number of hours saved on regression testing or the number of issues detected over a period of time, are relatively easy to calculate. However, you cannot impose an accurate monetary value on things like customer satisfaction, improved market positions, or the number of bugs that never appeared due to timely QA.
Advanced test automation ROI parameters
The basic formula for ROI calculation on automated testing (ROI = Cost savings / Investment cost) is almost universally applicable. However, its components may vary depending on your company. Furthermore, the metrics, cases, and best practices to pay attention to may also be different.
There are six parameters/variations of ROI calculations for automated testing, which may be stand-alone for your company or integrate into one bigger equation:
- Automation of new tests — this is the most standard case scenario, handled by the simplest “QA time saved” formula.
- Automation of older tests — I discussed regression testing, this is a best practice use case to seriously consider.
- All-around environment testing — knowing how your product handles in various environments may be key, considering most companies seek to diversify in this area. How much will you save on avoiding bugs across different environments?
- Defect leakage — the number of bugs “leaking” from development into production. Many experts consider this to be the most impactful parameter since it can do the most damage when affecting customers (and consequently, your business).
- Reuse or redundancy of tests — there’s little reason to build a new test when an existing one can be repurposed. Modular scripts are one way to mitigate this, but there are others. Consider the savings aspect here.
- Knowledge leakage — consider the average tenure of a QA engineer. Then think about what happens if they leave. What’s the cost of time spent on re-training a new one and re-engineering lost manual test cases? How can automation reduce your costs here?
“Yes, there are situations when experienced test engineers leave your company. The knowledge and experience regarding the testing framework, environment, or processes can be lost. Even when you hire a new tester, it doesn’t matter how efficient they are — picking up knowledge about how the testing process works in your company will take time. To prevent that, ensure that you have well-documented test processes, setup guides, and so on.”
Maxim Khymii, Automation QA Lead, TestFort
These can be parameters in your larger calculation or specific cases if you choose to implement automation in one area first. This leads us to the topic of the general best practices in the industry and how to ensure that the ROI formula actually corresponds to reality via correct tech integration.
Test automation implementation best practices
An important factor in calculating ROI and preparing to integrate a new company process is knowing the best practices of the industry, following which may directly or indirectly affect your costs and returns. The importance of introducing testing automation itself is not up for debate after all: with the average minute of downtime costing $5,600 and over 60% of software failures costing their owners more than $100,000, automation testing is one of the best options to ensure consistent functionality and stable performance. Here is how to do it the right way.
The two rules of thumb are: gradual introduction and long-term planning.
Calculating ROI on paper is one thing, but making sure that the company organically adopts a new technology is quite another. So, what are the three main points to consider during actual implementation?
- Don’t automate every single process right away
Rushing to automate every single test from day one is the other end of the extreme. As I mentioned before, manual testing has its place.
Consider carefully the pipeline of your automated transformation and what kind of balance your company has when it comes to testing needs. Which of the advanced ROI parameters listed above is the most vital for you? Where should you start?
- Understand that every test will become a regression test eventually
The danger of ignoring ROI on prior test automation is that it leaves a blind spot in your company: you’re not taking the future of your current tests into account.
The best practice for this is to integrate new tests as an element of your regression testing process right away.
- Evaluate the timeline (waterfall vs. shift-left model)
The shift-left testing method has been hailed as the saving grace of companies mired in expensive manual testing via the old waterfall model (where development and testing are done sequentially).
Shift-left is based on simultaneous development and testing via automation and agile practices, ensuring better quality and considerable time savings. In most cases, this is the go-to direction for most development and QA teams.
Conclusion
Automated testing has the potential to take most companies to a new level of efficiency. In today’s competitive markets, speed and quality of delivery are vital factors of survival and success.
The synergy between human expertise and automated tools is what defines the most dominant market players. Automation testing ROI calculations remain the main method of determining how to approach test automation, but managers must never forget that it’s not a cut-and-dry formula.
At the end of the day, your own internal resources and company structure, choice of automation partner, client needs, and other factors have a serious impact on your long-term returns. So, be cognizant of your company specifics, find trustworthy contractors, and value your team and development process health above all.
Top comments (0)