Test automation is generally introduced into a project to solve issues such as repetitive manual work, working with large data sets, or getting faster feedback in a CI/CD pipeline. With this buzz around test automation, you may be considering going forth and automating "all the things." You may have selected the testing framework and the team that would do the test automation. But have you seriously considered what you should automate or how feasible it is for you to automate everything?

Exhaustive testing is impossible. This is one of the key principles of software testing. While test automation significantly expands test scope and coverage, automating EVERY test takes more cost, effort, and time than is practical. Before any test automation begins, it is imperative for you to make a data-driven decision on what you will and will not automate --- and in what priority. When deciding what to automate, there are some basic items to consider, such as the automated test level at which you will be creating your tests. Automated Tests can be placed into three main levels: Unit Tests, Service/API Layer Tests, and User Interface Tests; you should evaluate the pros and cons of writing tests at each of these different levels before proceeding.

Regardless of which level of automated testing you perform, the objective should be to automate what brings the most value and impact, areas always tested for every release, and areas hard to test manually. You should also look at areas where there are huge amounts of data to be processed. All of these factors must be considered before automating. If you create automated tests blindly or on a whim, it increases the likelihood you will not get as much value from it as you should.

Detailed below are five (5) critical areas/factors to be aware of as you consider integrating automated testing into your current workflow.

Automate your smoke tests.

Ensuring that your application functions on a basic level is very important and should be done after each change or when a new build is made. This can be integrated into a CI/CD pipeline in order to verify that show-stopping bugs are caught early. Smoke tests ensure that the most important aspects and basic components of an application are functioning correctly. They test areas that always need to work and can help you decide whether a build or application is stable enough to proceed with further testing. Automating your smoke tests can:

1. Help to quickly detect show-stopping bugs early, such as app/webpage not loading, user unable to login, user not able to create an account, or payment services being down.
2. Enable faster troubleshooting of new and regression bugs. Smoke tests by nature have high coverage and shallow depth. This results in a wide variety of test cases being tested without going too deeply into the application. Once these critical areas are working, it allows you to go deeper into the application to test other features.
3. Result in less manual interaction and be time-saving. By integrating your automated tests into your Continuous Integration/Continuous Delivery (CI/CD) pipeline, your smoke tests are tested before a build is completed. This means that a build is not given to QA unless those smoke automated tests pass.

Automate tests that are always run, such as regression tests or tests always found at the beginning of every workflow.

This allows you to have a set of stable tests vital to the application. For example, creating a user account would need to be done before the user can login and view the dashboard, make payments, etc. By creating an automated regression suite, you can:

1. Run tests that quickly identify any bugs resulting from changes in the software. These changes may be new features or defect fixes.
2. Have a set of stable tests. Regression tests are normally for existing functionality, which means these functionalities are not new and have been tested multiple times. This prevents flaky automation tests, and you then must spend a lot of time manually testing since you are unsure whether the tests really failed or not.
3. Save time since you can now focus your manual testing efforts on more edge cases.

Automate extensive tests

Exhaustive tests such as inputting large sets of data, testing a form, or tests requiring the tester to do the same workflow multiple times but with a wide variety of data should be automated. You can create data-driven automated tests that can result in time being saved. For forms, automation allows you to quickly test different combinations of inputs to the form, such as whether fields are missing, incomplete, etc. Data-driven testing is very beneficial as it allows you to only modify the data instead of the test script to get different results. It is also very reusable and efficient.

Automate tests that require multiple configurations

This could include tests on different Operating Systems(OS) and Browser Combinations. This can be tedious to do manually and, if automated, can help save time. Automated tests can be run on different environments (such as Dev, QA, Staging, Integration, or PROD) by simply changing the environment variable. Tests can also be run in parallel, which reduces the time needed for execution. You can use different CI tools such as CircleCI to specify the OS, browser, and environments on which you want the parallel tests to be run. Ensure that you follow best practices when creating parallel tests in order to get as much benefit from these tests as possible.

Automate your performance tests.

This lets you avoid launch failures, bad user experiences, and performance regressions. Performance testing checks how the system performs under load and stress and also identifies bottlenecks. It checks the speed, response time, reliability, resource usage, and scalability of a software program under their expected workload. Automation can help you easily generate thousands of users to see how the application will respond. For example, Fandango is one of America's largest ticketing sites (with up to 36 million monthly movie-goers purchasing tickets from their site), and they wanted to ensure they were prepared for the movie Star Wars: The Last Jedi. They wanted to find out what their peak performance was and how to identify bottlenecks. QualityWorks automated their performance tests and provided them with reports that would help to identify bottlenecks and, as a result, they had a successful movie sale launch. This is something they can continue to use to ensure the performance of their website is up to a certain standard.

Your Next Step

This list of considerations is not exhaustive by any means but is a great starting point if you're considering how to integrate automation into your current QA workflow. Be strategic and focused in your approach. Sit with your current QA to discuss their current practices and to identify bottlenecks in their process that could be resolved with automation. Automating "all the things" is never the answer. Remember, before creating automated tests, think about the level you want to automate. Then, decide what tests you want to automate that would bring the most ROI for you and your team --- those that are repetitive, time-consuming, not flaky, critical to the business, or that test the performance of the application.

.........

About the Author

Julia is a Lead QA Consultant at QualityWorks with expertise in manual, automated, and API testing and training. She is passionate about sharing her knowledge and experience and contributes to the testing community through writing articles, and delivering testing content on Test Automation University as well as her Youtube Channel and blog. She also conducts testing Bootcamps for persons interested in entering the field of QA.

...........................................................................................................

Looking for ways to accelerate your testing process with automation? Reach out to our testing team at www.qualityworkscg.com