Introduction to mutation testing with PiTest

Mutation testing❓

Write tests for your software has many advantages:

• It ensures your business requirements are satisfied
• It's a security net against regressions
• If you practice TDD, it guides your implementation thanks to an emerging design. That's what we promote at Packmind since it helped us a lot.

Once we said that, the most challenging part of the works comes in: writing valuable and relevant tests.

It's one thing to say, "We have a bunch of unit tests and a high code coverage rate." But it's another thing to have relevant tests that do not run only to increase this metric beloved by non-technical people 😉

Mutation testing is a discipline that aims at improving the quality of our tests suite. Let's see the basic principles and a concrete example.

The concept of code mutation

Using a mutation testing framework will run many executions of your test suite with some modifications to your business code. These modifications are called mutants and can have the following form:

• An arithmetic (+, -, *, /, %) operator is changed to another value
• Some statements are removed.
• Boolean literals are reversed
• Assignment expressions are changed (+= to -=) for instance
• ...

The idea is straightforward: if your tests survive mutants' introduction, they're not robust enough. If your tests failed, the mutants have been killed ✅ !

A mutation testing framework will identify for you which mutants survived, questioning the relevance of these lines and also the scope of your tests.

An example with PiTest in Java

PIT is a mutation testing framework for Java. This is how you can include it your pom.xml file:

<build>
    <plugins>
        ...
        <plugin>
            <groupId>org.pitest</groupId>
            <artifactId>pitest-maven</artifactId>
            <version>1.7.2</version>
        </plugin>
    </plugins>
</build>

I'll take a straightforward example here to illustrate the concept.

package com.promyze.shop;

import java.util.ArrayList;
import java.util.List;

public class Basket {

    private List<Product> products = new ArrayList<Product>();

    public void addProduct(Product product) {
        this.products.add(product);
    }

    public int computePrice() {
        return this.products.stream()
                .mapToInt(Product::getPrice)
                .sum();
    }

    public int computePriceWithDiscount(int discount) {
        int price = computePrice();
        if (price - discount < 0) {
            return 0;
        }
        return price - discount;
    }
}

And this test suite was written with JUnit:

package com.promyze.shop.test;

import com.promyze.shop.Basket;
import com.promyze.shop.Product;
import org.junit.Test;

import static junit.framework.Assert.assertEquals;

public class BasketTest {

    @Test
    public void testProductPrice() {
        Basket basket = new Basket();
        basket.addProduct(new Product("orange", 10));
        basket.addProduct(new Product("apple", 5));

        assertEquals(15, basket.computePrice());
    }

    @Test
    public void testBasketPriceWithDiscount() {
        Basket basket = new Basket();
        basket.addProduct(new Product("orange", 10));
        basket.addProduct(new Product("apple", 5));

        assertEquals(12, basket.computePriceWithDiscount(3));
    }

    @Test
    public void testBasketPriceWithDiscountWhenDiscountIsHigherThanBasketPrice() {
       Basket basket = new Basket();
       basket.addProduct(new Product("apple", 2));

       assertEquals(0, basket.computePriceWithDiscount(3));
    }

}

Let's now run PIT with the following maven command:

mvn test pitest:mutationCoverage pitest:report -e -f pom.xml

⏲️ One important thing to know is that mutation testing may require a long execution time depending on the size of your codebase. You won't get an instant result, just be aware of that.

An HTML report is now available in the target/pit-reports folder, and you can navigate through the result. The report indicates which mutants were introduced and whether they were killed or survived (hover the Markers on the left side).

In this example, it probably changed "<" to "≤, " and the test still passed. I could indeed reinforce my test suite with another test or change that condition!

That's all for the intro; I think you got the concept now 🤗 !

Should I kill all mutants?

There's no silver bullet here since the fact that mutants survived may not imply an issue with your tests. You should review each mutant to figure out if an action is necessary.

To go further, you'll find a curated list of open-source mutation testing frameworks on this link. You can also discover another post we made on Property-based Testing.