This week, I set up a testing framework for my open source project, go-go-web.

For this I chose pytest, a very popular Python testing framework. I watched this getting started with pytest video by anthonywritescode, which gave a quick intro into setting up pytest.

pytest for Unit Testing

To install pytest, I simply ran this command in my VS Code terminal.

pip install -U pytest

Next I created a sample source code file and unit test file to try out pytest:

# src/sample.py
def square(x: float) -> float:
    return x * x

# tests/sample_test.py
from src import sample
def test_square():
    assert sample.square(5) == 25

Running pytest:

pytest
# Alternative
python -m pytest
# Run with detailed view
pytest -vv
# Run for one specific file
pytest <test file>.py
# Run showing print statements
pytest -s

The result:

Automate pytest for GitHub

To setup pytest to run on each GitHub code push / pull, see my post on CI Workflows here

Test with temporary files and directories

My app converts a .txt or .md file into .html. To test this core feature, I used pytest's tmpdir fixture which allows for testing using temporary directories and files. Here is a quick example of how it works:

# src/sample.py
def write_to_file(fname):
    with open(fname, 'w') as fp:
        fp.write('Hello\n')

# tests/sample_test.py
from src import sample
def test_write_to_file(tmpdir):
    file = tmpdir.join('output.txt')
    sample.write_to_file(str(file))
    assert file.read() == 'Hello\n'

Execute start up and clean up code before and after each test

You can also use fixtures to execute start up and clean up code that runs before and after each test function. Here is an example where I implement a fixture to run clean up code which reverts my CWD back to its original value.

import pytest

@pytest.fixture
def cleanup():
    # Startup code
    old_cwd = os.getcwd()   # Save the original CWD
    yield
    # Cleanup code
    os.chdir(old_cwd)   # Change CWD back to original

@pytest.mark.usefixtures('cleanup')
def test_some_function():
    ...

    # Change CWD to something else
    os.chdir(...)

    assert some_expression

Without the cleanup fixture above, if my assert failed there is no way for me to change the CWD back to its original value since pytest will exit from the test_some_function() function.

Test the console output

The pytest capfd fixture allows us to test what is printed out to the console. Here's an example:

# src/sample.py
def hello():
    print("hello")

# tests/sample_test.py
from src import sample
def test_hello(capfd):
    sample.hello()
    out, err = capfd.readouterr()  
    assert out == "hello\n"

Troubleshooting with pytest

Import error that only happens with pytest

pytest was getting an ImportError when running an import statement in one of my source code, even though that import statement has no issue running when I run the code outside pytest. Here is an example of the issue:

# src/sample.py
import utils as ut # <---- throws exception only with pytest
def hello():
    print("hello")

# tests/sample_test.py
from src import sample
def test_hello(capfd):
    sample.hello()
    out, err = capfd.readouterr()  
    assert out == "hello\n"

As you can see, the import statement throws an exception when run with pytest, but does not throw any exception when run normally with python src/sample.py:

After some digging online and trying a bunch of things, I found this solution:

Create a pytest.ini file in the root folder with the following contents. (src is the relative path from the root folder to your source code directory and tests is the path to your test files directory)

[pytest]  
pythonpath = src
testpaths = tests

An alternative solution would be to manually run the following code in the terminal before running pytest. This would have to be done every time you open a new terminal.

$env:PYTHONPATH = "<absolute path to your source code folder>"

Using venv

It's also beneficial to install pytest only inside a venv rather than globally just to avoid any potential issues or conflicts. If you get errors with pytest, you may want to try this:

1. Exiting and deleting any virtual env you are in
2. Uninstalling pytest with pip uninstall pytest
3. Recreating the venv with python -m venv .venv
4. Open a new VS Code terminal to enter the new venv
5. Inside the venv, run pip install pytest

Refactoring with pytest

It is possible to refactor pytest test code by creating a conftest.py file at the root. In this file, you can declare global variables, helper class and methods, and fixtures that can be accessed by all your test files.

Global variables

# conftest.py
def pytest_configure():
    pytest.var1 = 5
    pytest.var2 = 25

# tests/sample_test.py
from src import sample
def test_square():
    assert sample.square(pytest.var1) == pytest.var2

Helper class

# conftest.py
class Helpers:
    @staticmethod
    def file_contents(file_path):
        with open(str(file_path), "r", encoding="utf-8") as file:
            return file.read()

@pytest.fixture
def helpers():
    return Helpers

# tests/convert_test.py
from src import convert
def test_main(tmpdir, helpers):
    ...
    c.main(...) # this generates an output file test.html
    output_path = tmpdir.join('test.html')
    assert helpers.file_contents(output_path) == expected_contents

Check code coverage with coverage.py

Using coverage.py we can also check how much of our source code is covered by our unit tests.

Install: pip install coverage

Run: coverage run -m pytest

Reflection

While writing my test cases I learned a lot about how to use pytest. I got stuck a few times while trying to deal with the pytest import error and trying find ways refactor my code, and I have detailed my solutions in the above sections.

I also learned how to separate my core code into a main() function so that I could test it with pytest. I found a way to manually pass a list of arguments into my main function to mimic running my program on the command line with command line arguments.

For example, this test mimics the command line program call
python src/convert.py -o <tmp_output_dir> <tmp_input_file>

def test_main(tmpdir, helpers):
    arguments = c.parse_args([   
            "-o", 
            str(tmpdir), 
            helpers.new_file(tmpdir, "test.txt", 
                pytest.simple_txt_contents)
        ])
    # Run main function
    c.main(arguments)
    # Compare actual output file contents to expected
    output_path = tmpdir.join('test.html')
    assert helpers.file_contents(output_path) == 
        pytest.simple_html_from_txt

I also uncovered a bug which I would never have found otherwise. The only reason the bug was found was because the tmp files I created using pytest were on C:\ and my repo was located in F:\. If I was not using pytest to create tmp files, I may not have thought to test with files located on a different drive.

I learned a lot about pytest and unit testing and I hope this blog post will help someone with their unit testing. I may look into contributing to some unit test related issues in open source repos!