Saving time with contract testing

Hi everyone! Today's topic is Contract driven testing.

The way we spend our time is important.
It is important in our life and it is important for the company we work with.
I believe it's in our best interest to minimize time we invest in unproductive activities and maximize the one we spend working on building new ideas.

Ok, this is pretty general and someone would even say it's just common sense. What does it have to do with Contract testing?

Here I am gonna argue that in many circumstances, we can write unit tests instead of writing an end to end test or doing manual tests.
We can and should speed up the feedback loop (the amount of time needed to build confidence that the code we wrote actually works as expected), by testing the integration of different components as part of the unit tests suite.

Contract testing 101

When two entities communicate, changes to the supplier’s API can cause failures for all its consumers.

I have partially touched this topic when I discussed about Robustness Principle, indeed I think Contract testing is a way to make our services more Robust without giving up on development speed.

What do we do about it? We write integration tests/end to end tests/manual tests.
It is unquestionable that those kind of tests help to catch bugs before they end in production, but there are cons.

1. Running them requires setting up infrastructure, deploy the application, etc. They are way slower than unit tests, since this is the place where we call other services, we make network calls and use the database. Since we know that they are slow and require all the things ready, we cannot run them as often as we normally do with unit tests.
2. An implication of the first point is that they increase the dev feedback loop.
3. Since we share the same pipeline with other developers, a failing integration test does not necessarily mean we broke the pipeline. So, yet more time required to investigate what happened.

Before looking into a concrete example of contract tests, let's take a look at the test pyramid.

The pyramid is a graphical representation of how much testing of each type we should be doing. At the bottom we have unit tests which means in proportion we should write more unit tests than anything else:

• they make sure our code runs correctly when it runs in isolation
• they are easy to write and fast to run

We write lots of them and execute them every time we make a code change to the codebase or when we update one of our dependencies.

Based on what I said about integration tests and end to end tests, it's not a big surprise that they have been placed at the top of the pyramid.

Example

Let's see a concrete example of using contract tests instead of end to end test.

Example 1

Context: client to service communication
Scenario: A ReactJs application which manages a ToDo list for our users. The ToDo list is serialized and sent to a server, which stores the information to S3.
What we want to test: no change in the code causes a regression in the system, namely we are still able to deserialize the ToDo list we receive from the server and show it in a react component.

This is how the ToDo list might look like:

export class TodoList {
    items: Item[]

    constructor(items: Item[] = []) {
        this.items = items;
    }
}

// ......
// And this an item of our TodoList

export class Item {
    constructor(public description: string = '', 
                public isChecked: boolean = false) {}
}

Somewhere in the code we make an http request to get the TodoList, deserialize it and update the state of the view.

Approach 1 (Bad)

We could write an end to end test which:

• open the browser (using protractor, selenium or similar)
• request the react application
• add some items to the ToDo list
• save the ToDo list
• fetches the ToDo list again
• assert that the information in the ToDo widget is displayed correctly and no other error occurred.

This is exactly the kind of things we want to avoid to write; it is slow and fragile.
We can avoid the end to end tests by using contract testing.

Approach 2 (Good)

First let's create a contract for our ToDo list.
Tools we are going to use:

• Jest for unit testing (any other unit tests tool would work just fine)
• Typescript-json-schema to convert our to do list in a Json Schema
• Json schema validator to test that our ToDo list respects the contract

Let's define an utility function which create the contract the first time we run the test:

getOrCreateContract = (instance, filename) => {
    if (schemaDoesNotExist(filename)) {
        // TJS comes from the Typescript-json-schema lib
        const program = TJS.getProgramFromFiles([resolve(filename)], {}, basePath);
        const schema = TJS.generateSchema(program, instance, settings);
        saveSchema(CONTRACT_FOLDER, filename);

        return schema;
    }

    return getSchema(CONTRACT_FOLDER, filename);
};

The contract generated for our ToDo list looks like this:

{
    "$schema": "http://json-schema.org/draft-06/schema#",
    "definitions": {
    "Item": {
        "properties": {
            "description": {
                "default": "",
                    "type": "string"
            },
            "isChecked": {
                "default": false,
                    "type": "boolean"
            }
        },
        "type": "object"
    }
},
    "properties": {
    "items": {
        "items": {
            "$ref": "#/definitions/Item"
        },
        "type": "array"
    }
},
    "type": "object"
}

Now, let's write the contract test:

describe('ToDo List', () => {
    test('respect contract', () => {
        let todo = new TodoList([
            new Item('contract tests', true)
        ]);

        let contract = getOrCreateContract(todo, 'TodoList.schema.json');
        let contractValidator = new Validator();
        let respectContract = () => {
            contractValidator.validate(todo, contract);
        };

        expect(respectContract().error().length).toBe(0);
    });
});

This test gives us exactly the same confidence than we might get from the end to end test above, but it's way faster and does not need to communicate with the real dependencies.

Obviously, there are cases in which we want to update the contract. For example we could add a command line argument to override the existing contract.

Example 2

Context: service to service communication

I figured the post is getting a bit too long. The consequences of having a service to service communication do require some more concepts to introduce, so I'll write the example in a follow up post.

Conclusions

Integration tests have an important place in the test pyramid, but sometimes we overuse them.
Contract tests can save us time!

If you are interested in the topic, let me know!
This will help me understand if I should write a follow up post with service to service communication.

Thanks!
Nicola