Is your CI/CD pipeline slow? Do wait times make you feel unproductive? Parallel testing is an indispensable technique for reducing wait times. And mastering it is key to getting the most out of CI/CD.
What is Parallel Testing?
As evident as it may be, it bears repeating that testing must be automated. We can’t start a discussion about parallel testing without clarifying that it only makes sense when it is automated.
Our starting definition may look deceptively simple: we say we’re performing parallel testing when two or more tests are run simultaneously. Another way of demonstrating this is by showing a depiction of non-parallel testing. Take a look at this continuous integration pipeline.
Here tests are sequential, each step can only begin after the previous one is done. Even if every test takes a few seconds, the whole process can still amount to minutes. Waiting for a build is much more than an annoyance; it’s distracting and energy-draining.
Once we start using parallel testing, we’ll begin seeing something more along these lines:
By putting independent tests in parallel, we get rapid feedback and save precious minutes each time we make a change. Not only do we not lose focus on the problem we’re working on, we end up reclaiming many productive hours every week.
This is the power of parallel testing. Like upgrading your internet data plan or adding more lanes on a highway, it increases bandwidth — letting you do more work and getting farther faster.
When parallel testing is CI/CD’s best friend
The litmus test to determine the effectiveness of your pipeline is measuring its total run time. CI/CD is all about feedback loops — the sooner we have a result, the sooner we can fix, refactor, and iterate. When continuous integration takes more than 10 minutes or when continuous delivery has us waiting for more than 20 minutes, it’s high time we started looking into optimization.
Large testing suites are the low-hanging fruit and should be the starting point for optimization. We begin by identifying the longest-running test and checking if it’s possible to break it up into smaller, independent jobs. Then, we repeat the process until the pipeline is fast enough for our needs.
Tests that cannot be broken apart can sometimes be re-arranged so they don’t stand in the way of the other jobs.
What makes a test a good candidate for parallelization? Check its inputs and outputs. Do the tests generate something other processes need? What does it take to run these tests? The fewer dependencies a test has, the more likely it will work well in parallel.
What follows are a few typically good use cases for parallel testing.
Monorepos
Monorepos are code repositories containing many separate projects. As long as these projects are independent or loosely coupled, monorepos are a perfect fit for parallel testing.
When separation is not possible, when the projects are firmly interrelated, using parallelization becomes difficult. We can mitigate this problem with dedicated monorepo tools such as lerna or yarn.
Semaphore has first-class support for monorepos and can be configured to run tests only on the code that changes.
A variation of this theme happens when we have interrelated components, like a client and server in the same repository, which can be tested separately.
Static code analysis tests
Code analysis tests are another excellent candidate for parallel testing. Static tests represent the first line of defense in the quest to find errors in code. We find things like linters, coverage reports, and complexity analysis tools in this category. All of them efficiently run in parallel.
Testing for multiple environments and operating systems
We use the term environments in the most general way possible here, ranging from browsers to a mix of staging and production machines, and from a selection of mobile devices to different sets of data or API endpoints. The category can also include checking the application’s localization and internationalization features.
For example, testing an application for platforms such as iOS and Android is an excellent fit for parallelization.
The same applies to testing code on hybrid cloud environments.
Version and regression testing
Testing the code on various runtimes allows us to find compatibility errors. The following example uses a job matrix to run the tests in a combination of Java SDKs and application versions.
Benefits of parallelization
There are many advantages to using parallelization.
- Speed: faster builds translate instantly into doing more in less time. A faster testing cycle means that we can ship features quicker and release software more frequently.
- Faster time to recovery: this is how long it takes for the team to recover from a failure, such as a master branch breaking or a bad release. A parallel pipeline casts a broader net that lets us identify problems sooner, fix them, and release a fix quickly.
- Reduce bottlenecks: parallel testing is easy to scale into multiple environments. For instance, once we have tests on one version of Android, extending it to other versions shouldn’t impact overall pipeline runtime.
The limits of parallel testing
The bane of parallel testing is slow jobs. Even one abnormally slow job is enough to sink the total run time. The reason for this is that the pipeline cannot be faster than its slowest job, no matter how much parallelization we use. As Warren Buffet said: “You can't produce a baby in one month by getting nine women pregnant.”
The solution is to identify and analyze slow jobs to see if they can be optimized or broken down. To aid in this, Semaphore offers a test summary feature to help us analyze job outputs for many popular testing frameworks. And in many cases, it makes more sense to simply use a faster machine.
There are other caveats to keep in mind while working with parallel testing:
- Costs: parallel testing will result in higher resource utilization, which, when on a pay-per-use billing plan, will result in a bigger bill at the end of the month. That being said, the higher productivity obtained vastly outweighs the increased cost, so you may think of it as an investment. Our own research shows that a fast CI/CD pipeline provides a 41 to 1 return on investment.
- Complex dependencies: parallel testing becomes impractical when it’s impossible to separate components due to the nature of the project. Tightly coupled components are not generally recommended and the effort of uncoupling them is never wasted. But as long as there’s a high level of interdependence, it’s going to be hard to use parallel tests.
- Race conditions: when tests need to access the same external resources, such as an API or a database, we can run into race conditions or rate limits that give us false positives. This problem also can happen when test results are cached. In most cases, we can ensure race conditions cannot take place by rewriting the test logic.
- Optimization OCD: it’s possible to go overboard and run all tests in parallel, wasting resources while not getting a proportional gain in speed. Running fast and fundamental tests first allows us to get feedback on trivial errors in seconds while keeping the total CI/CD time in check.
- Platform limitations: your CI/CD platform might put a cap on parallelization. While Semaphore has no technical limits, there’s a safety quota that can be raised by submitting a support request.
- Flaky tests: flaky tests fail or succeed for no apparent reason. There may be many reasons for flakiness. It can, for instance, happen due to reliance on test order, lack of resources, or external dependencies. In any case, you just have to deal with them instead of relying on re-running them or ignoring false negatives as a short-term solution.
Horizontal vs. Vertical Parallelization
In general terms, we have two ways of scaling parallel tests in CI/CD: vertically and horizontally.
Vertical parallelization happens when using tools that can isolate tasks and run them concurrently, taking advantage of the many cores the CI machine has. Examples are Pants, Bazel, or Earthly.
Vertical tests are great because they are almost always very hands-off, and the tool will decide and automatically figure out the best way of running the tests. The flip side is that once we exceed the capacity of one machine, we have to figure out how to distribute the load in a testing cluster.
On the other hand, in horizontal parallelization, we explicitly configure each job and direct the execution flow by design. We’ll see a few examples of how this works in the next section.
Both kinds of parallelization work well on Semaphore. You can scale tests using vertical parallelization by choosing a more powerful machine. And, since Semaphore is a cloud-based service, you get instant, automatic horizontal parallelization.
How to Parallelize Testing in Semaphore
Semaphore supports four levels of vertical parallelization: job, block, pipeline, and workflow.
1. Parallel jobs
We use parallel jobs to break up a big task into more manageable chunks. For example, to run static analysis tools on the code or test different platform versions with a job matrix.
Job parallelization is the simplest form, and it’s just a matter of adding more than one job into the block.
Semaphore will run every job in the block simultaneously, using a separate and clean environment for each one.
2. Parallel blocks
Independent blocks can run in parallel, and can have their own parallel jobs. This is the second level of parallelization. Parallel blocks are used in monorepo setups and for testing independent components of an app. They are also helpful for testing different environments and running longer jobs outside of the main sequence of steps.
Semaphore automatically runs blocks in parallel when they don’t have dependencies.
3. Parallel pipelines
Parallelized pipelines are typically used to deploy onto multiple targets at the same time.
To run pipelines simultaneously, enable auto-promotions and set starting conditions to trigger continuous delivery or continuous deployment.
4. Parallel workflows
In the final level, we have workflow parallelization. A workflow is a group of pipelines triggered by a change in the repository. We can choose to run workflows in parallel or in series by assigning them to queues.
Semaphore uses queues for workflows, so configuring parallel workflow is vital to avoid waiting for pipelines while working on highly active repositories. To learn more about how Semaphore manages simultaneous workflows, check out the parallel pipelines page.
Note that you can only take advantage of parallelization in Semaphore if you’re on a paid, trial, or open-source plan. Free users are limited to one job at a time.
Tips for setting up parallel tests
Here are a few pointers on setting up parallel tests:
Fast tests first: keeping fast tests first helps us fail fast, while reducing time and the costs of running more long-lived tests. For instance, running static code tests first, then integration and end-to-end tests.
Tests with similar setup steps: use a block to run tests with similar preparation steps. For example, you may need to start a database to perform end-to-end tests and integration tests. You can run both in parallel in the same block.
Identify dependencies: block dependencies should mimic the internal dependencies of the project. For instance, it makes more sense to run all linting and unit tests first, in parallel, and then proceed to run more involved suites such as acceptance tests and integration tests.
Write parallelizable tests: having these concepts in mind while designing your test cases will help you write an optimal pipeline later.
Conclusion
Parallel testing lets us do more while waiting less. It’s an essential tool to keep sharp and ready so we can always establish a fast feedback loop. However, this isn’t an exact science, so when you start using parallel testing, ease in gradually and allow for a bit of trial and error to find the right balance for your project.
To continue reading about testing, check these great posts:
Top comments (0)