Code coverage vs. test coverage in Python

This article was originally written by Muhammed Ali on the Honeybadger Developer Blog.

If you have been writing tests for a while, you have probably encountered code coverage and test coverage. These concepts can be difficult to differentiate because they are somewhat intertwined. In this article, you will learn what code coverage and test coverage are all about, the basis of these concepts, and how to differentiate them. You will also learn the drawbacks and benefits of using them if you want to start applying them to future projects. Learning about these concepts will enable you to identify parts of your projects that have not been properly covered by test cases, which will, in turn, make your application more robust.

Generally, code coverage is objective; once your code is executed during a test, it is considered complete code coverage. However, test coverage is subjective and can be influenced by your consideration and scope. Keep reading for further explanation and examples.

Code coverage

When writing tests, as your project gets larger, it’s almost impossible to know whether all the parts of your codebase are covered by tests. The same limitation occurs when you want to know the percentage of your code that isn’t covered by the test and the actual code that isn’t covered. This is where code coverage comes in. Code coverage shows you the areas of your code that aren’t covered by tests, and with such information, you can investigate and find out how to fix it. It does so by checking the parts of the code executed during the testing process. It also provides you with a percentage of how much of your code has been covered by tests. Similar to how clay can be molded it into any form, to test code and get 100% coverage, you just need to mold an item.

Characteristics of code coverage

1. With code coverage, you can identify the part of your code that is not covered by a test, which makes writing tests easier.
2. It provides a percentage of the amount of code that has been tested.
3. With code coverage, when one value of the code feature is covered, the other possible values are neglected. Following our clay example, just molding a single item is enough to get 100%. It doesn’t take into account the other items which can be molded with clay.

Code coverage in Python

In this section, you will learn how to get code coverage for your Python code. We will first start by writing some Python functions and then write unit tests for them using the unittest module. Then, we will get code coverage with Coverage.py. You can install Coverage.py by running the following command:

pip install coverage

In the following code, the function sum_negative() adds only negative numbers and returns None otherwise. The sum_positive () function only adds positive numbers and returns None if they are negative.

To get this started, create a Python file and paste the following code:

def sum_negative(num1, num2):
    if num1 < 0 and num2 < 0:
        return num1 + num2
    else:
        return None

def sum_positive(num1, num2):
    if num1 > 0 and num2 > 0:
        return num1 + num2
    else:
        return None

Now we can write test cases for the code above using the unittest module. Create a new file name “tests.py” and paste the following code. The following code contains assertions that the functions output what is expected. There is one assertion for each return statement.

import unittest
from sample import sum_negative, sum_positive

class SumTests(unittest.TestCase):

    def test_sum(self):
        self.assertEqual (sum_negative(-5, -5), -10)
        self.assertEqual (sum_negative(5, 2), None)

    def test_sum_positive_ok(self):
        self.assertEqual (sum_positive(2, 2), 4)
        self.assertEqual (sum_positive(-5, -2), None)

The test cases above will give you a 100% code coverage. You can check by running the following commands.

coverage run -m unittest discover
coverage report -m

Although we are getting 100% code coverage here, the tests above are not well-rounded because they don’t test for other scenarios in which the code can be used.

Test coverage

Test coverage is a metric of how much of a feature in the code being tested is actually covered by tests. I know that it can be confusing, so I’ll use an analogy to illustrate. Then, we will use some code to make sure it’s clear. Taking our clay example, test coverage is implemented when you use the clay to build everything that can possibly be built with it. Here, the test that we did above that gave us 100% code coverage will be less when doing the test coverage evaluation. This is because many different things can be molded with clay, and they should also be considered when writing tests.

Characteristics of test coverage

1. It helps improve the quality of the code being covered by the test. This is because different scenarios in which that section of code can be applied are covered.
2. It makes your test coverage more robust.
3. There is a lot of manual work to be done since there is no tool for test coverage. Checking out the various ways in which your code can accept and send data can be very tasking.
4. It is more prone to errors since it is done manually.

Test coverage in Python

Unlike in code coverage where we only needed four assertions, here we will have more assertions. Using the sample code presented in the previous section, we have the following assertions:

import unittest
from sample import sum_negative, sum_positive

class SumTests(unittest.TestCase):

    def test_sum(self):
        self.assertEqual (sum_negative(-5, -5), -10)
        self.assertEqual (sum_negative(5, 2), None)
        self.assertEqual (sum_negative(5, 2), None) #new
        self.assertEqual (sum_negative(5, 2), None) #new

    def test_sum_positive_ok(self):
        self.assertEqual (sum_positive(2, 2), 4)
        self.assertEqual (sum_positive(-5, -2), None)
        self.assertEqual (sum_positive(5, -2), None) #new
        self.assertEqual (sum_positive(-5, 2), None) #new
        self.assertEqual (sum_positive(0, 0), None) #new

Conclusion

In this article, I covered what code and test coverage is about and how to differentiate between the two when working on a project. One thing you should know when it comes to coverage percentages, you should not be aiming to get 100% in test or code coverage because it doesn’t actually tell you how well-tested your program is. As I said earlier, if your code is tested with the wrong logic, it is still possible to get 100% coverage. As far as which you should use is concerned, it is up to you. If you are concerned about finding the parts of your code that have not been tested at all, code coverage will be your best bet. However, if you care about your test covering all possible scenarios, you should consider test coverage. Aside from that, a hybrid approach where both test and code coverage are used can also be employed to get the advantages of both.