Parallel test execution within the context of MSTestv2, NUnit3 and VSTest.Console

Keeping automated tests within a reasonable performance range is no trivial task and I'm always looking for ways to reduce the total duration of automated tests. Recently, I did some reading on the capabilities of a few different tools with regards to parallel execution of tests. These tools included MSTestv2 and NUnit3 when run with vstest.console.

While building a test case to compare the two I built two projects. One using MSTestv2 (v1.3.2) [DataTestMethod]s, designed to take advantage of DynamicData and another using NUnit3 (v3.10.1). Below is my MSTestv2 test.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Collections.Generic;
using System.Threading;

[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.MethodLevel)] //0 means use as many workers as possible

namespace mstest_parallel
{
    [TestClass]
    public class UnitTest1
    {
        private static IEnumerable<object[]> MyTestData()
        {
            for (int i = 0; i < 10; i++)
            {
                yield return new object[] { i * 1000 };
            }
        }

        [DataTestMethod]
        [DynamicData(nameof(MyTestData), DynamicDataSourceType.Method)]
        public void TestMethod1(int testVal)
        {
            Thread.Sleep(testVal);
        }
    }
}

Now, MSTestv2 has the ability to parallelize tests to some degree. Here are the parallelization options it offers at a high level:

• Class level - each thread executes a [TestClass] worth of tests. Within the [TestClass], the test methods execute serially
• Method level - each thread executes a [TestMethod]
• Custom - users can provide a plugin implementing the required execution semantics (Not yet supported).

These capabilities can be referred to as fine-grained parallelization features and the number of worker threads can be configured via assembly attribute [assembly: Parallelize(Workers = n, Scope = Execution.ClassLevel)] or a .runsettings file. More details here

Keep in mind these are different than MSTest v1's parallel options and also different from vstest.console's parallel options.

Vstest.console's parallel options focus on what its authors refer to as coarse-grained parallelization. This means one can parallelize the running of multiple test containers at once. So if you are running vstest.console from the command line and specify several test containers i.e., mytestlib1.dll mytestlib2.dll vstest.console will run each test container simultaneously up to the maximum possible on the machine or the maximum configured.

Why do we care? I want to squeeze as much performance out of the chosen test framework and runner as possible - ideally without having to write any threading code myself. Unfortunately, MSTestv2 (and v1) lacks a very specific and important parallelization option; The ability to run tests in parallel which are configured with [DynamicData] and/or [DataRow]. Evidence for this not being supported exists here, here and here. For our situation, this means our test method that utilizes [DynamicData] to generate unique cases will have all of the test cases per [TestMethod] run in serial rather than in parallel, thus making the majority of our parallelization efforts moot.

With parallelization turned on at the method level for the assembly our proof of concept MSTestv2 project showed the following results when using vstest.console to execute:

Total tests: 11. Passed: 11. Failed: 0. Skipped: 0.

Test Run Successful.

Test execution time: 47.4547 Seconds

Not that great. The majority of these tests are being run one by one.

NOTE: vstest apparently reports the 'base' test method as a test case when running MSTestv2 tests which is why the test count is 11 instead of 10. My understanding is that this does not actually run, but contains the overall results for all the tests associated with this [TestMethod].

Let's check out NUnit3. This framework offers the feature we want, which is running parameterized tests that have their cases generated at runtime in parallel.

The fundamental difference between the frameworks here is that MSTestv2 considers test cases via [DynamicData] and [DataRow] to all be under a single [TestMethod] whereas NUnit3 considers each test case its own separate [Test].

Here's our test method adjusted to be used with NUnit3:

using NUnit.Framework;
using System.Collections.Generic;
using System.Threading;

namespace nunit_parallel
{
    [TestFixture]
    [Parallelizable(ParallelScope.Children)] //parameterized tests are considered child tests
    public class UnitTest1
    {
        private static IEnumerable<object[]> MyTestData()
        {
            for (int i = 0; i < 10; i++)
            {
                yield return new object[] { i * 1000 };
            }
        }

        [Test]
        [TestCaseSource(nameof(MyTestData))]
        public void TestMethod1(int testVal)
        {
            Thread.Sleep(testVal);
        }
    }
}

The attribute specified at the top of the class [Parallelizable(ParallelScope.Children)] tells NUnit to parallelize every test method in the class and it considers each of our generated test cases a unique test method. Additionally, I am letting NUnit decide my [LevelOfParallism]. By default, NUnit3 uses the processor count or 2, whichever is greater.

Now, running with the same tests using vstest.console and same number of tests cases per [Test]:

Total tests: 10. Passed: 10. Failed: 0. Skipped: 0.

Test Run Successful.

Test execution time: 17.5729 Seconds

Predictably, the time to run the suite of tests is drastically shorter.

One important part of this setup is how these tests are run. Vstest.console handles running tests from each test framework for us as long as we can specify the necessary test adapter. For both NUnit3 and MSTestv2, the adapters are available as NuGet packages. Adding these NuGet packages to your test project will cause the necessary libraries to be copied out upon build and vstest.console will find them for you so long as they are in the same directory as the test containers you specify.

I hope this helps explain some of the options regarding parallelizing tests, especially parameterized ones. I learned a ton while studying this stuff and am I sure this will continue to be the case while writing these types of test suites. If you believe anything I have written above needs to be corrected please reach out to me.