How I Set up My Python Workspace

TL;DR: In this tutorial, I’ll teach you what I do to have multiple Python versions and tools installed without conflicts.

I’ve been working with Python for the last 5 years. In the beginning academically and as a hobby, then professionally. One thing that has always been very common is having multiple projects in different Python versions. Regardless of the language you use, having projects targeting at distinct versions can be a nightmare. When it comes to Python, it’s no different. As a matter of fact, it’s worse. For instance, there are libraries that only work with Python 2, despite not being supported anymore. Fortunately, there are a couple of tools that can help us have seamless experience handling different Python versions.

This guide will focus on Linux, but you can easily adapt to macOS. In fact, I have a MacBook at work and follow the same steps to configure my workspace with little modifications. Also, I use zsh as my default shell, but this guide applies to bash as well.

So, without further ado, here's how I setup my Python workspace:

Step 1. Define your requirements

Before getting into the actual configuration, you must take some time to define what your optimal setup looks like. For me, it looks like this:

1. It must have Python 2.7+, Python 3.6, Python 3.7, and Python 3.8 installed.
2. Python 3.8 must be the default version.
3. I must be able to switch between versions easily.
4. I must be able to fire up a ipython using the default python version I specified.
5. Dependency tools I use - pipenv and poetry - must work with all Python 3+ versions.

Step 2. Installing and configuring multiple Python versions

To install multiple versions and being able to switch between them, I use pyenv.
pyenv has several benefits such as:

• Lets you change the global Python version on a per-user basis.
• Provides support for per-project Python versions.
• Everything is installed in the $HOME directory. This means no risk of messing up the default Python installation.
• Supports pypy, anaconda, CPython, Stackless-Python and others!

In addition to pyenv, I also use the pyenv-virtualenv plugin to manage my virtual environments.

Installing pyenv and its plugins

Since last year, I've been using Homebrew both on macOS and Linux. So, the following steps are OS agnostic for me.

brew install pyenv
brew install pyenv-virtualenv

Creating a directory for the virtualenvs

Each project should have each own virtual environment associated with. This way we can work on more than one project at a time without introducing conflicts in their dependencies. I keep all my virtualenvs under the $HOME/.ve directory.

mkdir -p $HOME/.ve

Once the folder is created I add the path to my ~/.zshrc (for bash users, add it to ~/.bashrc):

cat <<"EOT" >> ~/.zshrc
# pyenv config
# Set virtualenv dir
export WORKON_HOME=~/.ve
# Initialize pyenv
if command -v pyenv 1>/dev/null 2>&1; then
  eval "$(pyenv init -)"
fi
# Initialize pyenv-virtualenv
eval "$(pyenv virtualenv-init -)"
EOT

The variable WORKON_HOME tells pipenv where to place your virtual environments.

Once that's done, you can restart your shell by either closing and opening a new window or running:

exec $SHELL

Installing all Python versions I need.

PY_DEFAULT=3.8.5
PY_VERSIONS=( $PY_DEFAULT 3.7.8 3.6.11 2.7.18 )

for py_version in "${PY_VERSIONS[@]}"
do
    echo -e "Installing Python $py_version...\n\n"
    # Install specific Python version
    pyenv install $py_version
done

Step 3. Installing the tools

For dependency management I use two different tools, pipenv, and more recently poetry. Eventually I will probably settle on poetry by at the moment I need both.

Also, I rely a lot on jupyter notebooks, for quick data analysis tasks; and ipython as a fancy Python interpreter.

Again, I don't want to pollute the global installation. To avoid that we can use pyenv-virtualenv to create virtualenvs for the tools.

Let's install jupyter, pipenv, and poetry.

# Creating tools3 venv
pyenv virtualenv $PY_DEFAULT tools3

# Activating
pyenv activate tools3

# Upgrade pip
pip install --upgrade pip

# Install Jupyter
pip install jupyter

# Install Jupyter extensions
pip install jupyter_nbextensions_configurator rise
jupyter nbextensions_configurator enable --user

# pipenv
pip install pipenv

# poetry (using preview version to fix a bug in the 1.0.10 version)
pip install --pre poetry -U

If you use pipenv, make sure to install its completion script. You can add it to your shell like so:

echo 'eval "$(pipenv --completion)"' >> ~/.zshrc

And for poetry:

# Oh-My-Zsh
mkdir -p $ZSH/plugins/poetry
poetry completions zsh > $ZSH/plugins/poetry/_poetry

Since the completion is a oh-my-zsh plugin, I need to add poetry to the plugins list in my ~/.zshrc file.

plugins(
    poetry
    ...
)

This can be accomplished using sed:

sed -i.bak 's/^plugins=(\(.*\)/plugins=(poetry\n        \1/' ~/.zshrc

Now, we need to configure poetry to create virtualenvs inside ~/.ve, just like pipenv.

echo 'export POETRY_VIRTUALENVS_PATH=$WORKON_HOME' >> ~/.zshrc

More poetry configurations

One more thing, pipenv's default behaviour is to load any environment variables defined in the .env in the root of the project. It does that on two occasions, when running pipenv shell and pipenv run. poetry, on the other hand, does not support that. As a workaround, I need to load it manually.

To automate the process, I created a shell function that loads it whenever I run poetry shell or poetry run. If I want to disable it, I can set the POETRY_DONT_LOAD_ENV variable. This is, again, similar to how PIPENV_DONT_LOAD_ENV works.

cat <<"EOT" >> ~/.zshrc
# Get the poetry's full path
POETRY_CMD=$(which poetry)
# Allow poetry to load .env files
function poetry() {
    # Define the full command. i.e. poetry [run|shell|version]
    POETRY_FULL_CMD=($POETRY_CMD "$@")

    # if POETRY_DONT_LOAD_ENV is *not* set, then load .env if it exists
    # also, only loads when for "run" and "shell" commands.
    if [[ -z "$POETRY_DONT_LOAD_ENV" && -f .env && ("$1" = "run" || "$1" = "shell") ]]; then
        echo 'Loading .env environment variables…'
        env $(grep -v '^#' .env | tr -d ' ' | xargs) $POETRY_FULL_CMD
    else
        $POETRY_FULL_CMD
    fi
}
EOT

Now deactivate the virtualenv tools3.

# Deactivating the venv
pyenv deactivate

Step 4. Setting interpreters priority

Now that we have all the versions we need installed we must establish some sort of priority. Basically, I want to use poetry, or any other tool, without activating the virtualenv where I installed them. We can do that using pyenv global command.

PY_DEFAULT=3.8.5
PY_VERSIONS=( $PY_DEFAULT 3.7.8 3.6.11 2.7.18 )
pyenv global $PY_VERSIONS tools3 system

Let's check:

$ pyenv versions
* system (set by /home/miguel/.pyenv/version)
* 2.7.18 (set by /home/miguel/.pyenv/version)
  2.7.18/envs/tools2
* 3.6.11 (set by /home/miguel/.pyenv/version)
* 3.7.8 (set by /home/miguel/.pyenv/version)
* 3.8.5 (set by /home/miguel/.pyenv/version)
  3.8.5/envs/tools3
* tools3 (set by /home/miguel/.pyenv/version)

Everything looks good, it's time to restart the shell and that's it.

Preventing accidental installation of packages

One thing that happened a lot to me was installing packages inside one of the global interpreters using pip. As I mentioned earlier, it's a good idea to keep each Python interpreter intact. If we need to install anything, I prefer to create a new virtualenv, like I did for tools3.

Now, how can we lock each Python installation?

That's actually pretty straightforward. We can set pip to only install packages if there's a virtualenv active.

echo 'export PIP_REQUIRE_VIRTUALENV=true' >> ~/.zshrc

What about pipenv and poetry?

Now, if I have a poetry project that requires Python 3.6, I tell poetry to use the 3.6 version and it will automatically create a virtualenv using the Python 3.6.11 I installed.

Example:

# pyproject.toml 
...
[tool.poetry.dependencies]
python = "~3.6"
...

Output:

$ poetry env use 3.6   
Creating virtualenv sandbox-aBhl6cgV-py3.6 in /home/miguel/.ve
Using virtualenv: /home/miguel/.ve/sandbox-aBhl6cgV-py3.6
$ poetry shell        
Loading .env environment variables…
Spawning shell within /home/miguel/.ve/sandbox-aBhl6cgV-py3.6
$ . /home/miguel/.ve/sandbox-aBhl6cgV-py3.6/bin/activate
(sandbox-aBhl6cgV-py3.6) $

Conclusion

That’s pretty much it! I hope this tutorial is useful for you, just as it’s for me. Whenever I need to reconfigure my workspace, I follow those steps. Also, I’ll probably create a shell script to automate it instead of copying and pasting from this guide. This tutorial was initially based on https://medium.com/@henriquebastos/the-definitive-guide-to-setup-my-python-workspace-628d68552e14, which served as inspiration on how to setup my own workspace.

How I Set up My Python Workspace first appeared on miguendes's blog.