Improve our code with the property-based testing and fast-check

🧪 Why property-based testing?

At Packmind, we're convinced that methodologies such as Test-Driven Development (TDD) can really improve the quality of our code. But developers need relevant inputs to ensure our code answers a business requirement. Behavior-Driven Development or Example Mapping sessions help to generate concrete examples of the behavior of a feature. This clearly brings value for developers and allows them to start writing tests and business code.

However, our tests usually cover a restricted range of inputs that come from our examples. We assume they are representative enough for our tests. But what if we want to test edge cases? Or if we're not sure that our test is representative?

This is where property-based testing comes in. This post is a short intro to this concept.

❓ What is PBT?

Property-based testing aims to identify and test invariants: predicates that should be true, whatever the input data are. An invariant ****is a business rule that can be written as a predicate.

A PBT framework will generate random input data and check if the invariant is valid. If one single execution fails, this means the code under test may have some defects in its implementation. But the PBT framework will give you the input data, meaning you can reproduce the problem.

PBT is not something new, the first research studies on that topic are from 1994, from Fink & Levitt.

🎓 Illustration with a test case

Our platform Packmind is designed for best coding practices sharing. Users can merge two practices if they have the same intention. Each practice can have zero, one, or multiple categories. During a merge, categories of both practices should be merged into the target practice.

Here is a simple implementation of this business rule (the mergePractice method) and a unit test written in JS with Mocha and Chai. For simplicity, the code is written in the same file.

const chai = require('chai');

var expect = chai.expect;

function mergePractice(practice1, practice2) {
    return {
        categories: practice1.categories.concat(practice2.categories)
    };
}

describe('Merge two practices', function() {
    it('should merge categories of the source practices into the target practice', function() {
        const practice1 = { categories : ["JS"]};
        const practice2 = { categories : ["Node"]};
        const targetPractice = mergePractice(practice1, practice2);
        expect(targetPractice.categories).to.eql(["JS", "Node"]);
    });
});

As you can see, this code does not handle any edge case. The implementation is clearly straightforward with a concat operation. We consider only a typical case with two practices having a single category.

🚀 An implementation of PBT with fast-check

Now comes the PBT with fast-check, an open-source framework developed by Nicolas Dubien for Javascript and Typescript.

From our example, we can identify two invariants during the merge of two practices:

• There can't be any duplications in the categories of the target practice.
• The categories of the target practice should not contain elements that do not appear in the source practices' categories.

Here is the implementation of the two predicates.

const noDuplicationsPredicate = (targetPractice) => {
    if (!targetPractice.categories || !targetPractice.categories.length) {
        return true;
    }
    var uniqueCategories = new Set(targetPractice);
    return uniqueCategories.size === targetPractice.categories.length;
}

const targetCategoriesShouldNotContainCategoriesNotInSourcePractices = (sourcePractice1, sourcePractice2, targetPractice) => {
    if (targetPractice.categories.length) {
        return targetPractice.categories.every(c => 
            sourcePractice1.categories.includes(c) || sourcePractice2.categories.includes(c));
    }
    return true;
}

And here is the test with fast-check :

describe('Merge two practices', function() {

    it('should merge categories of the source practices into the target practice based on random data', function() {
        fc.assert(
            fc.property(
                fc.array(fc.string()),
                fc.array(fc.string()),
                (cat1, cat2) => {
                    const practice1 = { categories : cat1};
                    const practice2 = { categories : cat2};
                    const targetPractice = mergePractice(practice1, practice2);
                    return noDuplicationsPredicate(targetPractice) && 
                            targetCategoriesShouldNotContainCategoriesNotInSourcePractices(practice1, practice2, targetPractice)
                }),
            { numRuns: 10000 })
    });
});

In short: fc.assert runs the property, fc.property defines it, and fc.array(fc.string()) generates a random array of string values, possible empty. We ask here to run 10,000 iterations of the test. Of course the documentation of the framework will give your more information!

After execution of our test (running with Mocha), we got the following error:

1) Merge two practices
       should merge categories of the source practices into the target practice:
     Error: Property failed after 1 tests
{ seed: 1361468347, path: "0:0:3:2:3:2", endOnFailure: true }
Counterexample: [[],["",""]]

See the counterexample? You could argue that such a case should not normally happen, as categories should never be null or empty, but this is not the point here, it's just an illustration of PBT :). We've got which inputs raise the error.

Thanks to that, we can slightly edit our business function:

function mergePractice(practice1, practice2) {
    return {
        categories: practice1.categories.concat(practice2.categories).filter(c => c && c !== "")
    };
}

Next run, we realized that our business case did not cover duplicated categories :

Error: Property failed after 161 tests
{ seed: -759575891, path: "160:1:0:4:6", endOnFailure: true }
Counterexample: [["!"],["!"]]

The mergePractice method should now be updated to avoid duplications.

I think you've got it, right? PBT can complement your existing tests, and we showed an example of how this can help improve our codebase quality and the robustness of our tests.

There are many more exciting features, such as Shrinking, which tries to simplify the understanding of a failing test by reducing the problem at its lowest level.

In a future post, we'll discuss how mutation testing can also be relevant to improve our codebase.