The size of our testing team at Audacia has grown dramatically in recent years, tripling in size since 2019. This has led to a large increase in available resources. QAs are now dedicated to one or two projects and have time to look at Automated regression, security and performance testing. Senior Software Tester at Audacia, Cameran Pullan, discusses the benefits of performance testing in our latest insight.
In the past, manual testing has been the main focus within the company. This is because the size of the test team meant we did not have available resources to dedicate to non-functional testing. The recent growth has allowed us to change the way that the test team works.
We have adopted an automation first approach to testing. This means that we are writing automation tests from the inception of the project, which allows an entire regression test to be run automatically. Saving time on manual testing gave us the opportunity to look at two key areas of non-functional testing; Performance and Security.
Performance Testing
Imagine a scenario where a project is nearing completion and the client drops the bombshell that performance is of paramount importance to their business and customers.
This non-functional requirement was not mentioned during the scoping of the system requirements. The developers have created the application under the assumption that performance is not a high priority. And the test team have accepted the application as functional.
In a typical project with no test automation, the real user load is only tested at the point of going live. We may be aware of the expected number of users, but have not tested the system with that volume.
The system fails on the first day it goes live due to a larger number of users than the system had previously dealt with. This failure renders the application unusable for the client.
Who is at fault for this?
The short answer is: everyone. Someone should have questioned this at all points of the development process.
Living in a Perfect World
In a perfect world the performance of an application should be considered at all points of the development lifecycle: Design, Implementation and Testing. However, this is an ideal that is rarely achieved due to the time and budget constraints of software development.
Where this is not possible, we aim to effectively test the application so we have an awareness of performance bottlenecks in the system.
Even when performance testing is considered at the start of the project, it is not usually actioned until the end of the project due to time constraints.
Integrating Performance Testing into Lifecycles
At Audacia we aim to run key forms of testing — Manual, Security, Performance and Accessibility — at all points in a project’s life. This starts with scoping out the non-functional requirements and continues with testers planning & writing a full stack of tests.
Since taking an automated first approach to testing, we have seen an improvement in the overall quality in products. Time saved on regression and other manual testing has allowed most staff to spend a larger portion of time on non-functional tests. These are becoming a core activity within the test team and give us more confidence in our products.
The fact that we can create performance tests at the beginning of the project means we are aware of potential performance issues much earlier in the process.
Locating and fixing performance issues through the life of the project encourages the whole team to adopt the mindset that performance is important. Having that mindset in place makes performance a top consideration when creating products. With this consideration, everyone has greater confidence in their work when the project goes live.
Holistic Strategy
When your software testing team has an appreciation for the importance of application performance it becomes easy to get quite single-minded with the performance requirements. For example the team only considers page load times for a public facing application.
While this approach will improve performance it does not capture the whole user experience and can leave the team vulnerable to unexpected performance issues.
We Should Take a Holistic Approach to Testing
Further Reading: 5 Best Practices
For example, alongside page load times, we should test how easy/quickly a user can get meaningful data. It would be unacceptable for the page to load in a few seconds, but requests to get meaningful data take over a minute.
Manufactured Chaos
Your software testing team could integrate performance testing into all points of your lifecycle. Also your team could consider all predictable failures and protect against these failures. This will help with your application roll-out failing or if there are unforeseen issues.
Chaos by its very definition is the inability to predict future events.
How Do We Plan for the Unexpected?
Inspired by Netflix, we manufacture our own chaos and include it in the software lifecycle.
Netflix created a number of open-source tools (for example Chaos Monkey) that prepare developers and testers to work in an artificially chaotic environment. The overall purpose of these tools is to randomly shut down infrastructure such as servers, Load balancers etc.
While this may seem strange, at first, it means that project teams are constantly battling unforeseen circumstances. This changes the way that the team thinks; they inherently create a more resilient application by creating added protection against chaos.
Netflix’s tools work best for solutions with a large infrastructure i.e. globally used software. We can then take these principles and apply them in a way that is more suitable for the project size and scope.
On small projects this could be putting a small module in the software that randomly fails API requests at random intervals.
Working in a predictably chaotic environment creates a mind-set that can anticipate chaos and create enhanced protection against it.
Expect the Unexpected
The reality of the software industry is that it is chaotic due to ever-changing requirements and priorities. If you manufacture your own chaos it allows your team to be better prepared for actual unplanned events. This model will encourage your team to have a changed mindset and improve their quality assurance skills. Long term this leads to a better product being delivered to the client.
Top comments (0)