How to Measure the Effectiveness of Unit Tests Using Code Coverage

We sometimes talk about code coverage when discussing the quality of our unit tests. This week we’ll look at what code coverage is, how we can measure it, and its usefulness.

What is Code Coverage?

Code coverage measures the amount of production code that a test suite goes through while running. It’s often expressed as a percentage that shows the ratio of code that’s tested, against the amount that existing tests can’t or won’t. We can use a few different metrics when describing coverage, but we’ll focus on two for the remainder of this article: line coverage, and branch coverage.

Getting Started

If you’re using Visual Studio 2022 Enterprise, you’ll find a built-in feature in the Test menu. But if you’re using other editions (or prefer not to use it), there are many alternatives – users of tooling from JetBrains might have dotCover, and there are free options too.

With over 80,500 installations at the time of writing, Fine Code Coverage is one such option (I have no association with it, by the way). It’s packaged as a Visual Studio extension, meaning you can easily install it via the Manage Extensions option within Visual Studio. After doing so, you should find a new dockable tool window that looks like Image 1. (If not, you should be able to find it under View > Other Windows > Fine Code Coverage.)

It’ll be blank to begin with but will display a report once your projects’ tests have run. Image 2 shows an example of what it might look like.

Understanding the Results

The two columns that visually stand out most are headed Line coverage, and Branch coverage. The bars represent the corresponding percentage values in the columns to their left and offer a quick and simple way to interpret the results.

Line Coverage

Line coverage describes the number of code statements the tests have run through. When reading this, it’s important to remember that a value of 100% doesn’t imply an absence of bugs. Imagine we have the following classes:

public class DataObject
{
    public string Name { get; set; }
    public string Description { get; set; }
}

public class DataObjectFactory
{
    public DataObject CreateDataObject(
        string name, string description)
    {
        var obj = new DataObject();
        obj.Name = name;
        obj.Description = "";

        return obj;
    }
}

In DataObjectFactory, we want calling CreateDataObject to return a new DataObject instance with its Name and Description set to their corresponding arguments. However, there’s a bug that sets Description in the returned DataObject instance to an empty string: despite having 100% line coverage, the following test fails.

[Test]
public void CreateDataObjectPopulatesAllFields()
{
    // Arrange

    var factory = new DataObjectFactory();

    // Act

    var obj = factory.CreateDataObject("name", "description");

    // Assert

    Assert.That(obj.Name, Is.EqualTo("name"));
    Assert.That(obj.Description, Is.EqualTo("description"));
}

Line coverage doesn’t reflect the amount of tested logic – only the amount that is run through while testing. As such, we shouldn’t try to influence it just for the sake of scoring higher. It’s better to have a low but realistic score, than to have it artificially boosted by engineering tests to simply run through code. There’s no prize for scoring 100% when some tests have no validation purpose, and doing so may lead to a false sense of security.

By having a low score, we know we might need to do something about it to increase the overall amount of tested code. We should also keep in mind that unit testing is just one form of testing. Having low line coverage might not matter if the areas lacking coverage are tested by other means, e.g. a separate suite of end-to-end tests.

Branch Coverage

Branch coverage describes the number of unique logic branches passed through while testing. To explain this, consider the following code sample; it’s a simple function that converts a bool value into its string equivalent.

public class BoolToStringConverter
{
    public string Stringify(bool input)
    {
        if (input)
        {
            return "true";
        }
        else
        {
            return "false";
        }
    }
}

In the following test we only use the logic on one branch: when input is true. As the if statement in Stringify has two potential outcomes, our branch coverage is 50%.

[Test]
public void StringifyReturnsInputAsString()
{
    // Arrange

    var converter = new BoolToStringConverter();

    // Act

    var result = converter.Stringify(true);

    // Assert

    Assert.That(result, Is.EqualTo("true"));
}

To achieve 100% branch coverage, we’d need to run the test twice – once with true for input, and once with false:

[TestCase(true, "true")]
[TestCase(false, "false")]
public void StringifyReturnsInputAsString(
    bool input, string expected)
{
    // Arrange

    var converter = new BoolToStringConverter();

    // Act

    var result = converter.Stringify(input);

    // Assert

    Assert.That(result, Is.EqualTo(expected));
}

Summary

Code coverage is a way to estimate how much of a project its tests cover. But you should interpret the results with caution to avoid a false sense of security.

There are many tools both paid-for and free that can help you to measure coverage. Two important metrics are line coverage, and branch coverage.

Line coverage describes how much code is run through while testing. Remember that a high percentage doesn’t mean bug-free code. Conversely, a low percentage might not matter if the lacking areas are tested by other means.

Branch coverage offers a different measurement. Instead of recording the number of statements used while running your tests, it describes how many distinct logical pathways were taken.

By analysing the results, you’ll be able to refine your testing strategies. And by using different methods to complement each other, you’ll be able to catch more bugs before they go to production.

Thanks for reading!

This article is from my newsletter. If you found it useful, please consider subscribing. You’ll get more articles like this delivered straight to your inbox (once per week), plus bonus developer tips too!