DEV Community

Cover image for Analyzing and enforcing .NET code coverage with coverlet
Daniel Genezini
Daniel Genezini

Posted on • Originally published at blog.genezini.com on

Analyzing and enforcing .NET code coverage with coverlet

#dotnet #csharp #testing #devops

This post is part of a series:
Part 1 - Defining formatting rules in .NET with EditorConfig
Part 2 - Enforcing .NET code style rules at compile time
Part 3 - Analyzing and enforcing .NET code coverage with coverlet

Introduction

Automated software tests are a requirement for ensuring we are delivering a product with quality to our users. It helps in finding bugs and requirements not fulfilled at development time, but also decreases the cost of maintenance by making the future changes to our code safer. Besides, the act of writing testable code alone increases the quality of the code we are writing because testable code has to be decoupled.

In this last post of this series, I'll show how to analyze and enforce a minimum code coverage in our applications, and how to use integration tests to increase our testing surface.

What is code coverage?

Code coverage is a software metric that shows how much of our code is executed (covered) by our automated tests. It is shown as a percentage and can be calculated with different formulas, based on the number of lines or branches, for example. The higher the percentage, more of our code is being tested.

Analyzing the code coverage of our application

In this example, we have an ASP.NET Core API with a simple use case class that checks an input number and returns a string telling if the number is even or odd:

using CodeCoverageSample.Interfaces;

namespace CodeCoverageSample.UseCases;

public class IsEvenUseCase : IIsEvenUseCase
{
    public string IsEven(int number)
    {
        if (number % 2 == 0)
        {
            return "Number is even";
        }
        else
        {
            return "Number is odd";
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

For now, we have only one unit test for this use case class:

using CodeCoverageSample.UseCases;

namespace CodeCoverageSample.UnitTests;

public class IsEvenUseCaseTests
{
    [Fact]
    public void EvenNumber_ReturnsEven()
    {
        //Arrange
        var isEvenUseCase = new IsEvenUseCase();

        //Act
        var result = isEvenUseCase.IsEven(2);

        //Assert
        Assert.Equal("Number is even", result);
    }
}
Enter fullscreen mode Exit fullscreen mode

To analyze the code coverage of our application, first we need to install Coverlet's MSBuild integration using the coverlet.msbuild nuget package in our test project:

Install-Package coverlet.collector
Enter fullscreen mode Exit fullscreen mode

Then, run the dotnet test command with the Coverlet options on the solution or project folder:

dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=../TestResults
Enter fullscreen mode Exit fullscreen mode

We are using the following options:

  • CollectCoverage: Inform dotnet test to use coverlet to collect the code coverage data;
  • CoverletOutputFormat: The format of the report that coverlet will generate (opencover, cobertura, json). More here;
  • CoverletOutput: The path where the coverage report will be saved in. This path is relative to the test project;

This will print the code coverage result in a table and generate a report file named TestResults.opencover.xml:

Code coverage results on the command line

⚠️ We can also run coverlet on the command line with the coverlet.collector nuget package, but it has limited options and doesn't print the results in the command line. More details here;

Generating HTML reports

Coverlet generates the report in formats that are not easily readable by humans, so we need to generate an HTML report based on Coverlet report. To do it, we'll usa a tool called ReportGenerator.

Installing ReportGenerator

ReportGenerator is installed as a .NET global tool.

To do this, we run the following command:

dotnet tool install --global dotnet-reportgenerator-globaltool --version 4.8.6
Enter fullscreen mode Exit fullscreen mode

Generating an HTML report of the opencover report

To generate an HTML report based on a Coverlet report, we run the following command:

reportgenerator "-reports:TestResults.opencover.xml" "-targetdir:coveragereport" -reporttypes:Html
Enter fullscreen mode Exit fullscreen mode

We are using the following options:

  • reports: The path to the coverage report;
  • targetdir: The path where the HTML report will be saved in;
  • reporttypes: The format the report will be generated in.

The command output will tell the relative path to the generated report: coveragereport\index.html.

ReportGenerator output

Opening the coveragereport\index.html file we can see the project Line and Branch coverage:

HTML report summary

Clicking on CodeCoverageSample.UseCases.IsEvenUseCase we can see details of the code coverage by method (in the table) and the line and branch coverage for the class:

Details of a class on the HTML report

Line vs Branch coverage

But what is line coverage and branch coverage?

  • Line coverage: Indicates the percentage of lines that are covered by the tests;
  • Branch coverage: Indicates the percentage of logical paths that are covered by the tests (if, else, switch condition, etc).

In the example below, we can see that two lines in the else branch are not covered by the tests.

This will result in:

  • 50% of branch coverage, because only the if branch is covered;
  • 71.4% of line coverage, because only 5 of the 7 lines are covered.

Example of a coverage report

Code coverage on Visual Studio

The is an extension called Run Coverlet Report that integrates Coverlet and ReportGenerator with Visual Studio.

  1. First, we need to install the coverlet.collector nuget package in our test projects. Xunit template already has this package installed by default. 
Install-Package coverlet.collector
Enter fullscreen mode Exit fullscreen mode
  1. Then, navigate to Extensions > Manage extensions and install the Run Coverlet Report extension.

Manage Extensions screen

  1. Navigate to the new option Tools > Run Code Coverage. This will generate the ReportGenerator HTML report that will be open in Visual Studio.

Code coverage report inside Visual Studio

Also, after running the code coverage tool, Visual studio will read the coverlet report and show the coverage in our source file:

Code inside Visual Studio showing code coverage

Improving our code coverage

Fixing the unit tests

Now we will implement the OddNumber_ReturnsOdd method to test the logical path we didn't test before:

using CodeCoverageSample.UseCases;

namespace CodeCoverageSample.UnitTests;

public class IsEvenUseCaseTests
{
    [Fact]
    public void EvenNumber_ReturnsEven()
    {
        //Arrange
        var isEvenUseCase = new IsEvenUseCase();

        //Act
        var result = isEvenUseCase.IsEven(2);

        //Assert
        Assert.Equal("Number is even", result);
    }

    [Fact]
    public void OddNumber_ReturnsOdd()
    {
        //Arrange
        var isEvenUseCase = new IsEvenUseCase();

        //Act
        var result = isEvenUseCase.IsEven(3);

        //Assert
        Assert.Equal("Number is odd", result);
    }
}
Enter fullscreen mode Exit fullscreen mode

This will increase our average coverage to 50% of branch and 22.58% of line:

Code coverage increased

And 100% for the IsEvenUseCase class:

Code coverage showing all lives covered for the IsEvenUseCase class

Implementing integration tests

Integration tests using the WebApplicationFactory class (More here) are also considered in the code coverage reports. Let's look at our IsEvenController and Program classes coverage:

Let's implement a simple integration test. It will just instantiate our API and make a call passing a number and validate the results:

using Microsoft.AspNetCore.Mvc.Testing;
using System.Net;

namespace CodeCoverageSample.UnitTests.IntegrationTests;

public class IsEvenIntegrationTest : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly WebApplicationFactory<Program> _factory;

    public IsEvenIntegrationTest(WebApplicationFactory<Program> factory)
    {
        _factory = factory;
    }

    [Theory]
    [InlineData(2, "Number is even")]
    [InlineData(3, "Number is odd")]
    public async Task Number_ReturnsCorrectAndOk(int number, string expectedResult)
    {
        var HttpClient = _factory
            .CreateClient();

        //Act
        var HttpResponse = await HttpClient.GetAsync($"/iseven/{number}");

        //Assert
        Assert.Equal(HttpStatusCode.OK, HttpResponse.StatusCode);

        var ResponseStr = await HttpResponse.Content.ReadAsStringAsync();

        Assert.Equal(expectedResult, ResponseStr);
    }
}
Enter fullscreen mode Exit fullscreen mode

Now we run the code coverage report again and the IsEvenController and Program classes are covered by the tests:

Removing code from the code coverage analysis

If we want to remove a class or method from the code coverage analysis, we can decorate it with the ExcludeFromCodeCoverage attribute:

using System.Diagnostics.CodeAnalysis;

namespace CodeCoverageSample;

[ExcludeFromCodeCoverage]
public class DoNotTestMe
{
    ...
}
Enter fullscreen mode Exit fullscreen mode

ℹ️ We can also create custom attributes to exclude from coverlet code coverage. Details here.

Ignoring auto-properties

Coverlet has the SkipAutoProps option to exclude the auto-properties from the coverage report.

For example, this class doesn't have any logic and doesn't need that the get and set methods of its properties be tested:

namespace CodeCoverageSample;

public class NoLogicHere
{
    public int Id { get; set; }
    public int Name { get; set; }
}
Enter fullscreen mode Exit fullscreen mode

Just set the SkipAutoProps to true when running the code coverage from the command line:

dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=TestResults -p:SkipAutoProps=true
Enter fullscreen mode Exit fullscreen mode

⚠️ Unfortunately, the Run Coverage Report extension still doesn't allow us to configure the coverlet parameters. There is an open pull request with this feature awaiting for approval here.

Enforcing a minimum code coverage on the build pipeline

Just like code style and code quality rules, that I talked about in my previous post, we need to enforce a minimum code coverage in our build pipeline to maintain a level of quality in our code. Coverlet has the Threshold option that we can set to a minimum percentage and it will fail the tests if our code coverage is below this percentage:

dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=TestResults -p:SkipAutoProps=true -p:Threshold=80
Enter fullscreen mode Exit fullscreen mode

We can also use the ThresholdType option to set the type of coverage to enforce. Not specifying will enforce all types of coverage (Line, Branch and Method). Details here.

Example of a failed test with Threshold=80

Liked this post?

I post extra content in my personal blog. Click here to see.

Follow me

References and Links

Top comments (0)

Read next

teo_nordic_d6988ee4c1221e profile image

Exploring Java's Role in Cloud Computing and AI for 2024

Teo Nordic -

siddhantkcode profile image

When k6 eats up your RAM: Slashing memory usage in load tests

Siddhant Khare -

greenfieldcoder profile image

Adding Revision Support to Entities in Entity Framework Core

Jimmy -

satya_prakash profile image

A Beginner’s Guide to Component Testing with Playwright

satyaprakash behera -