Testing

Testing can only prove the presence of bugs, not their absence. — Edsger W. Dijkstra

There, it should work now. — All programmers

Above quotes chosen from this collection at softwareengineering.stackexchange.

General tips

Another crucial aspect in the programming journey is knowing how to write tests. In bigger projects, usually there are separate engineers (often in much larger number than developers) to test the code. Even in those cases, writing a few sanity test cases yourself can help you develop faster knowing that the changes aren't breaking basic functionality.

There's no single consensus on test methodologies. There is Unit testing, Integration testing, Test-driven development (TDD) and so on. Often, a combination of these is used. These days, machine learning is also being considered to reduce the testing time, see Testing Firefox more efficiently with machine learning for an example.

When I start a project, I usually try to write the programs incrementally. Say I need to iterate over files from a directory. I will make sure that portion is working (usually with print() statements), then add another feature — say file reading and test that and so on. This reduces the burden of testing a large program at once at the end. And depending upon the nature of the program, I'll add a few sanity tests at the end. For example, for my command_help project, I copy pasted a few test runs of the program with different options and arguments into a separate file and wrote a program to perform these tests programmatically whenever the source code is modified.

assert

For simple cases, the assert statement is good enough. If the expression passed to assert evaluates to False, the AssertionError exception will be raised. You can optionally pass a message, separated by a comma after the expression to be tested. See docs.python: assert for documentation.

# passing case
>>> assert 2 < 3

# failing case
>>> num = -2
>>> assert num >= 0, 'only positive integer allowed'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: only positive integer allowed

Here's a sample program (solution for one of the exercises from Control structures chapter).

# nested_braces.py
def max_nested_braces(expr):
    max_count = count = 0
    for char in expr:
        if char == '{':
            count += 1
            if count > max_count:
                max_count = count
        elif char == '}':
            if count == 0:
                return -1
            count -= 1

    if count != 0:
        return -1
    return max_count

def test_cases():
    assert max_nested_braces('a*b') == 0
    assert max_nested_braces('a*b+{}') == 1
    assert max_nested_braces('a*{b+c}') == 1
    assert max_nested_braces('{a+2}*{b+c}') == 1
    assert max_nested_braces('a*{b+c*{e*3.14}}') == 2
    assert max_nested_braces('{{a+2}*{b+c}+e}') == 2
    assert max_nested_braces('{{a+2}*{b+{c*d}}+e}') == 3
    assert max_nested_braces('{{a+2}*{{b+{c*d}}+e*d}}') == 4
    assert max_nested_braces('a*b{') == -1
    assert max_nested_braces('a*{b+c}}') == -1
    assert max_nested_braces('}a+b{') == -1
    assert max_nested_braces('a*{b+c*{e*3.14}}}') == -1
    assert max_nested_braces('{{a+2}*{{b}+{c*d}}+e*d}}') == -1

if __name__ == '__main__':
    test_cases()
    print('all tests passed')

max_count = count = 0 is a terse way to initialize multiple variables to the same value. Okay to use for immutable types (will be discussed later) like int, float and str.

If everything goes right, you should see the following output.

$ python3.9 nested_braces.py
all tests passed

As an exercise, randomly change the logic of max_nested_braces function and see if any of the tests fail.

assert statements can be skipped by using python3.9 -O

Writing tests helps you in many ways. It could help you guard against typos and accidental editing. Often, you'll need to tweak a program in future to correct some bugs or add a feature — tests would again help to give you confidence that you haven't messed up already working cases. Another use case is refactoring, where you rewrite a portion of the program (sometimes entire) without changing its functionality.

Here's an alternate implementation of max_nested_braces(expr) function from the above program using regular expressions.

# nested_braces_re.py
# only the function is shown below
import re

def max_nested_braces(expr):
    count = 0
    while True:
        expr, no_of_subs = re.subn(r'\{[^{}]*\}', '', expr)
        if no_of_subs == 0:
            break
        count += 1

    if re.search(r'[{}]', expr):
        return -1
    return count

pytest

For large projects, simple assert statements aren't enough to adequately write and manage tests. You'll require built-in module unittest or popular third-party modules like pytest. See python test automation frameworks for more resources.

This section will show a few introductory examples with pytest. If you visit a project on PyPI, the pytest page for example, you can copy the installation command as shown in the image below. You can also check out the statistics link (https://libraries.io/pypi/pytest for example) as a minimal sanity check that you are installing the correct module.

# virtual environment
$ pip install pytest

# normal environment
$ python3.9 -m pip install --user pytest

After installation, you'll have pytest usable as a command line application by itself. The two programs discussed in the previous section can be run without any modification as shown below. This is because pytest will automatically use function names starting with test for its purpose. See doc.pytest: Conventions for Python test discovery for full details.

# -v is verbose option, use -q for quiet version
$ pytest -v nested_braces.py
=================== test session starts ====================
platform linux -- Python 3.9.0, pytest-6.2.1, py-1.10.0,
    pluggy-0.13.1 -- /usr/local/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/learnbyexample/Python/programs
collected 1 item                                           

nested_braces.py::test_cases PASSED                  [100%]

==================== 1 passed in 0.03s =====================

Here's an example where pytest is imported as well.

# exception_testing.py
import pytest

def sum2nums(n1, n2):
    types_allowed = (int, float)
    assert type(n1) in types_allowed, 'only int/float allowed'
    assert type(n2) in types_allowed, 'only int/float allowed'
    return n1 + n2

def test_valid_values():
    assert sum2nums(3, -2) == 1
    # see https://stackoverflow.com/q/5595425
    from math import isclose
    assert isclose(sum2nums(-3.14, 2), -1.14)

def test_exception():
    with pytest.raises(AssertionError) as e:
        sum2nums('hi', 3)
    assert 'only int/float allowed' in str(e.value)

    with pytest.raises(AssertionError) as e:
        sum2nums(3.14, 'a')
    assert 'only int/float allowed' in str(e.value)

pytest.raises() allows you to check if exceptions are raised for the given test cases. You can optionally check the error message as well. The with context manager will be discussed in a later chapter. Note that the above program doesn't actually call any executable code, since pytest will automatically run the test functions.

$ pytest -v exception_testing.py
=================== test session starts ====================
platform linux -- Python 3.9.0, pytest-6.2.1, py-1.10.0,
    pluggy-0.13.1 -- /usr/local/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/learnbyexample/Python/programs
collected 2 items                                          

exception_testing.py::test_valid_values PASSED       [ 50%]
exception_testing.py::test_exception PASSED          [100%]

==================== 2 passed in 0.02s =====================

The above illustrations are trivial examples. Here's some advanced learning resources:

• realpython: Getting started with testing in Python
• pytest — calmcode video series and Testing Python Applications
• obeythetestinggoat — TDD for the Web, with Python, Selenium, Django, JavaScript and pals
• testdriven: Modern Test-Driven Development in Python — TDD guide and has a real world application example
• Serious Python — deployment, scalability, testing, and more