Chapter 5: Testing Multiple Environments with TOX.

• This chapter is part of the series:
  

  Understanding the Python Tox Ecosystem

  Namah Shrestha ・ Oct 21 '22

  #python #tox #testing #pytest
• This tutorial builds upon the project structure from Chapter 3 and Chapter 4. I suggest understanding Chapter 3 and Chapter 4 before moving forward in order to understand the project structure.
  • Chapter 3:
    

    Chapter 3: Using PyTest, MyPy and flake8 for testing

    Namah Shrestha ・ Oct 21 '22

    #python #testing #pytest #flake8
  • Chapter 4:
    

    Chapter 4: A Quick tour of PyTest

    Namah Shrestha ・ Oct 21 '22

    #python #pytest #testing #unittesting

• So far, this is the structure of our code.
  

  - src/
      - something/
          - __init__.py
          - py.typed
          - app.py
  - tests/
      - __init__.py
      - conftest.py
      - test_app.py
  - .gitignore
  - LICENSE
  - pyproject.toml
  - README.md
  - requirements.txt
  - setup.cfg
  - setup.py
  
  

• Table of contents:

  • 5.1 - The Need for Tox and the Tox configuration file
  • 5.2 - Understanding the Tox environments and variables
  • 5.3 - Understanding the flow of Tox environments
  • 5.4 - Executing TOX

5.1 The Need for Tox

• The code so far has run on our local environment.
• How do we know that the code runs on multiple environments? Since we are building packages, we should know which versions of python can install our package and run it.
• This is where Tox comes in. To work with Tox we need a tox.ini file which is Tox's own configuration file.

• Let’s consider the following tox.ini file in the root directory of the project.
  

  [tox]
  minversion = 3.8.0
  envlist = py36, py37, py38, py39, flake8, mypy
  isolated_build = true
  
  [gh-actions]
  python =
      3.6: py36, mypy, flake8
      3.7: py37
      3.8: py38
      3.9: py39
  
  [testenv]
  setenv =
      PYTHONPATH = {toxinidir}
  deps =
      -r{toxinidir}/requirements_dev.txt
  commands =
      pytest --basetemp={envtmpdir}
  
  [testenv:flake8]
  basepython = python3.6
  deps = flake8
  commands = flake8 src tests
  
  [testenv:mypy]
  basepython = python3.6
  deps =
      -r{toxinidir}/requirements_dev.txt
  commands = mypy src
  

• To read more about tox file structure, we can read: https://tox.wiki/en/latest/config.html.

5.2 Understanding the Tox environments and variables

• We can read about all of this from [tox.wiki](http://tox.wiki) , https://tox.wiki/en/latest/config.html.

• All global settings are defined in the [tox] section. In our case,
  

  [tox]
  minversion = 3.8.0
  envlist = py36, py37, py38, py39, flake8, mypy
  isolated_build = true
  

  • minversion is the minimum version of tox-library required to parse this tox file.
    • 3.8.0 is the minimum version of tox-library required to parse this tox file.
  • envlist is for determining the environment list that toxis to operate on and so on.
  • There are many other global setting variables that are available in the tox.wiki website. We do not need to understand everything as of now.

• Test environments are defined under the testenvsection and individual testenv:NAMEsections, where NAMEis the name of a specific environment. This section also has a lot of options to set. For our case, we will go through:

  • basepython: Name or path to a Python interpreter which will be used for creating the virtual environment.
  • deps: Environment dependencies. Installed usually from requirements file or manually written.
  • commands: The commands to be called for testing. Only execute if [commands_pre](https://tox.wiki/en/latest/config.html#conf-commands_pre) succeeds. commands_pre is another option we have not talked about.
  • setenv: Each line contains a NAME=VALUE environment variable setting which will be used for all test command invocations.

• We might also notice {toxinidir} which is a variable inbuilt in tox. We can learning more about changing its working directory: https://stackoverflow.com/questions/52503796/change-tox-workdir.

  • For now, we just need to understand that: from tox global settingssection from tox documentation, the .tox directory which is working dir, is created in directory where tox.iniis located.
  • Turns out tox also creates a working directory just like git does.

• We also have {envtmpdir} which is another variable inbuilt in tox.

  • {envdir} basically points to the virtualenv directory that tox creates.
  • If we do {envdir}/tmp it points to the tmp directory inside the virtualenv.
  • This is what {envtmpdir} points to. Basically {envtmpdir} = {envdir}/tmp.
  • It will be cleared each time before the group of test commands is invoked. In our case, we have three group of test commands: a [test] which is inbuilt in tox and two test:NAME which are our custom test groups.
  • There are other similar variables: {envlogdir}={envdir/log}. It defines a directory for logging where tox will put logs of tool invocation.

• tox-gh-actionsis a tox plugin which helps running tox on GitHub Actions with multiple different Python versions on multiple workers in parallel.

5.3 Understanding the flow of Tox environments

• Tox allows us to create a bunch of different virtual environments, install our package into those environments and then run the test groups on each of those environments.

• The first four py36, py37, py38, py39, are built-in versions of python that tox already knows about.
  

  [tox]
  minversion = 3.8.0
  envlist = py36, py37, py38, py39, flake8, mypy
  isolated_build = true
  

• Their configuration goes in the testenv block and the tests are run in those environments with the following commands.
  

  [testenv]
  setenv =
      PYTHONPATH = {toxinidir}
  deps =
      -r{toxinidir}/requirements_dev.txt
  commands =
      pytest --basetemp={envtmpdir}
  

  • We set the PYTHONPATH to {toxinidir}. From the previous section we know it equals to PYTHONPATH=<top level of project directory>. Since, we know toxinidir points to wherever the tox.ini file is located. In our case, it is located at the top level of the project directory.
  • As you can see, in testenv , we have set deps=-r{toxinidir}/requirements_dev.txt. From the previous section we know it equals to deps=-r <project_directory>/requirements_dev.txt. Since, we know toxinidir points to wherever the tox.ini file is located. In our case, it is located at the top level of the project directory.
  • We install the requirements and then run the pytest --basetemp=<virtualenv>/tmp command in all of the environments passed along. Since, we know that {envtmpdir} points to <virtualenv>/tmp directory.

• Then we have the last two environments: flake8, mypy.
  

  [tox]
  minversion = 3.8.0
  envlist = py36, py37, py38, py39, flake8, mypy
  isolated_build = true
  

• flake8 and mypy are not built-in environments in tox. Therefore, we have to create separate testenv:NAME block for them. Which is why we added:
  

  [testenv:flake8]
  basepython = python3.6
  deps = flake8
  commands = flake8 src tests
  
  [testenv:mypy]
  basepython = python3.6
  deps =
      -r{toxinidir}/requirements_dev.txt
  commands = mypy src
  

• Here, testenv:flake8 is the flake8 virtual environment and testenv:mypy is the mypy virtual environment.

• Since flake8 and mypy are not versions of python and only commands that we want to run, we need to specify which version of python we want to run on. We do this on basepython.

• deps again are all the dependencies. We can specify individual dependencies or we can specify the requirements file.

• commands is the actual command that runs in these blocks.

• We also have, gh-actions block which we will discuss in the next chapter.

5.4 Executing TOX

• Once we have the tox.ini file and the tox command.
• To get the tox command, pip install tox.

• We can now just run the tox command and everything will be automated.
  

  $ tox
  

• When we run it, we notice that it takes very long. Why is that?

  • When we run the tests locally, it runs on our virtual environment.
  • But like we discussed, tox creates a new virtual environment, install everything into that and run tests in all of those environments.

• It is advised to run tox only before commit and push. Otherwise, just use pytest , to save time during development.