Parallelise Your NUnit Tests to Be More Productive and Waste Less Time

We previously touched on how NUnit runs tests sequentially by default. While individual tests typically don’t take too long, they all contribute to the total time required to run the whole set. When run frequently, the time spent waiting adds up.

In this week’s article, we’ll explore running tests in parallel. First, we’ll look at how we mark tests to run concurrently. Then we’ll look at some issues that can cause unexpected behaviour when doing so, and how we can get around them.

The More the Merrier

In NUnit, we use the Parallelizable attribute to indicate which tests we want run at the same time. The following example shows the easiest way to set all tests in a fixture to run in parallel. Here, we’re applying it at the class level; by passing ParallelScope.All as an argument, we’re saying we want all tests within it to start running as soon as possible.

[Parallelizable(ParallelScope.All)]
public class MyTests
{
    ...

However, we can also be more precise; we have other options that let us specify exactly what’s allowed to run together. For example, at the other end of the spectrum, we might want to allow only a select number of tests to run in parallel. To do this, we’d apply the attribute to individual tests (instead of their fixture) and pass ParallelScope.Self as its argument.

But if parallelising tests can save us time, why wouldn’t we want all tests to run in this way?

The short answer is compatibility: some tests might not have been written with parallelism in mind and could have unpredictable behaviour.

The Problem with Shared Variables and State

Ideally, every test should be fully self-contained. When shared variables and state are introduced, there’s a potential for problems to arise.

Setup Methods

After writing a few tests, we might start noticing common code fragments appearing in multiple tests. We typically want to write the cleanest code we can; experience has shown me that doing otherwise can come back to bite when returning to the code after some time. To this end, we might want to apply the DRY principle and refactor to avoid duplication. Where the repeated code occurs in the initial Arrange sections of tests, it’s tempting to use setup attributes to declare and/or reset shared variables at class (or greater) scope: this would minimise the code in these sections.

However, we need to be careful when running tests in parallel. Let’s assume we can run up to two tests at once, we have three tests to run in total, and each one uses class-scoped variables that are reset at the beginning of each test by a setup attribute. For simplicity, we’ll refer to our tests as:

• Test A
• Test B
• Test C

The following sequence of events would cause our results to become unreliable.

1. Test A and Test B start running.
2. Test B finishes running.
3. Test C starts running, resetting the class-scoped variables.
4. Test A finishes, but its results have been affected by step (3).

Depending on the test, an alternative to shared variables and setup methods could be using factory methods when setting up test components.

Non-concurrent Collections

Local collections can be used as test-substitutes for databases and other data stores. When used in multiple tests, declaring them at class-level can make sense. Suppose we have the following interface.

public interface IMyRepository
{
    public string GetValue(string key);
    public string SetValue(string key, string value);
}

In our tests, we might decide to mock it. To make it behave appropriately, we could add callbacks that use a Dictionary to back it.

public class MyRepositoryTests
{
  private readonly IDictionary<string, string> _databaseSubstitute
    = new Dictionary<string, string>();

  [Test]
  public void MyRepositoryTest()
  {
    // Arrange

    var repository = new Mock<IMyRepository>();

    repository
      Setup(r => r.GetValue(It.IsAny<string>()))
        .Returns<string>(key => _databaseSubstitute[key]);

    repository
      .Setup(r =>
        r.SetValue(It.IsAny<string>(), It.IsAny<string>()))
      .Callback<string, string>((key, value) =>
        _databaseSubstitute[key] = value);

    // Remainder of test code...
  }
}

This would work without problems under single-threaded usage. But running multiple similar tests in parallel could result in unexpected behaviour when doing anything other than reading – not all operations on Dictionary can be run safely from multiple threads at the same time.

To work around this, we could replace Dictionary with ConcurrentDictionary. However, this also brings additional complexity. A simpler alternative would be to make _databaseSubstitute a local variable in each test that uses it.

Test Order

One final consideration is the order tests are run in. It’s possible to specify this with the Order attribute; when unspecified, tests (appear to) run in alphabetical order. Consider the following fixture. (This was written to show the effect of running tests in parallel rather than verifying code.)

public class TestOrder
{
    private readonly IList<int> _results = new List<int>();

    [Test]
    [Order(0)]
    public void Test1()
    {
        _results.Add(1);
    }

    [Test]
    [Order(1)]
    public void Test2()
    {
        _results.Add(2);
    }

    [Test]
    [Order(2)]
    public void Test3()
    {
        _results.Add(3);
    }

    [Test]
    [Order(3)]
    public void TestResultsCollection()
    {
        Assert.That(_results[0], Is.EqualTo(1));
        Assert.That(_results[1], Is.EqualTo(2));
        Assert.That(_results[2], Is.EqualTo(3));
    }
}

All tests pass when run only on a single thread:

1. Test1 is run first, adding 1 to the list of results.
2. Test2 is run when Test1 completes, adding 2 to _results.
3. Test3 runs after Test2. This adds 3 to _results.
4. Finally, TestResultsCollection is run. As steps 1-3 happened sequentially, the order of the integers in _results is as expected.

We can choose to run this fixture in parallel by applying [Parallelizable(ParallelScope.All)] at the class level. However, TestResultsCollection will have inconsistent results: it will sometimes pass, and sometimes fail.

To understand why, let’s assume we have the capacity to run all four tests simultaneously. It’s impossible to predict exactly when each test will start running and complete – this will be dependent on many things including how long each test takes to run, the operating system’s thread scheduling system, and overall system load from background processes at the time.

The test fixture is essentially a concurrent system without thread-synchronisation. In other words, we can’t rely on Test1 always writing to the first position in the list, Test2 the second, and Test3 the third. We can partially address the situation by writing results to a Dictionary instead of a List; the order of the first three tests will no longer matter:

[Parallelizable(ParallelScope.All)]
public class TestOrder
{
    private readonly IDictionary<string, int> _results =
        new Dictionary<string, int>();

    [Test]
    [Order(0)]
    public void Test1()
    {
        _results["Test1"] = 1;
    }

    [Test]
    [Order(1)]
    public void Test2()
    {
        _results["Test2"] = 2;
    }

    [Test]
    [Order(2)]
    public void Test3()
    {
        _results["Test3"] = 3;
    }

    [Test]
    [Order(3)]
    public void TestResultsCollection()
    {
        Assert.That(_results["Test1"], Is.EqualTo(1));
        Assert.That(_results["Test2"], Is.EqualTo(2));
        Assert.That(_results["Test3"], Is.EqualTo(3));
    }
}

However, there’s a chance that the assertions in TestResultsCollection will be encountered before some of the other tests complete. The simplest and safest way to avoid this is to rewrite the tests to isolate each one so they don’t rely on the results of others – either that, or removing the Parallelizable attribute.

Summary

Running tests in parallel can reduce time otherwise spent waiting for them to complete. This is simple in NUnit, but you must ensure your tests are compatible to prevent unexpected behaviour.

Tests should be fully self-contained. While it’s tempting to use setup attributes to reduce code duplication, a new test starting can interfere with already-running tests if shared variables are altered.

It’s also important to remember not all data collections are fully thread safe. Performing write operations from different threads can cause exceptions to be thrown.

Finally, if the test runner has capacity to start a new test, it will do so independently of the status of other tests. If a test relies on others completing in a predefined order, it may need rewriting.

Thanks for reading!

This article is from my newsletter. If you found it useful, please consider subscribing. You’ll get more articles like this delivered straight to your inbox (once per week), plus bonus developer tips too!