Code Coverage - more than just a number

Code coverage

Code coverage is a set of tools, that examine the percentages of code covered by the tests.
I’m working with Jest on daily basis, in this article I will use examples from this test framework.

To run coverage report, you can use following command:

  jest --coverage

You can see the results right away in your terminal or open nicer GUI with coverage in src/coverage/lcov-report/index.html

The metrics

• lines - simply how much of you code is taking part in the tests
• statements - same as above but it counts only instructions
• functions - how many functions were called during test
• branches - all branches of if/else, switches, conditional flows

Lets look at silly example:

// Test case

describe('isErrorRoute - helper to determine if url from input is recognized as error route in routing', () => {
  it('should', () => {
    expect(isErrorRoute('https://example.com/error')).toEqual(true);
    expect(isErrorRoute('https://example.com/error/not-found')).toEqual(true);
  });
});

// Implementation

export const isErrorRoute = (url?: string) => {
  if (!url) {
    // 1. branch not covered
    // 2. both 'statements' in line 4 and 5 also not covered
    console.log('statement');
    return false;
  }

  return url.includes('/error');
};

And the test with following metrics

• lines - 60% (3/5)
• statements - 66.66% (4/6)
• functions - 100% (1/1)
• branches - 0% (0/1)

What information code coverage gives us ?

• what parts of application are not covered with tests at all
• acts as a general guide for further analysis, where there might a be a real need of adding tests

What information code coverage won’t give you ?

• how much behaviour is covered
• it tell you nothing about overall tests quality
• covering code with tests ≠= covering functionality with tests

The code coverage requirements

Most of the times there is requirement of having 80% code coverage in order to add new functionality to the codebase.

While I can understand the purpose of it, there is a huge chance, that some developers may focus solely on meeting the 80% threshold, without considering the real purpose of testing.

Common gotchas to spot on for “80% threshold focused” devs:

• Over-reliance on mocks and stubs
• No assertions in the tests
• Writing many very similar tests
• Ignoring code paths that are difficult to test (which may be a crucial parts for functionality)
• Testing implementation details rather than behavior