TL;DR: In this tutorial, I’ll teach you what I do to have multiple Python versions and tools installed without conflicts.
I’ve been working with Python for the last 5 years. In the beginning academically and as a hobby, then professionally. One thing that has always been very common is having multiple projects in different Python versions. Regardless of the language you use, having projects targeting at distinct versions can be a nightmare. When it comes to Python, it’s no different. As a matter of fact, it’s worse. For instance, there are libraries that only work with Python 2, despite not being supported anymore. Fortunately, there are a couple of tools that can help us have seamless experience handling different Python versions.
This guide will focus on Linux, but you can easily adapt to macOS. In fact, I have a MacBook at work and follow the same steps to configure my workspace with little modifications. Also, I use zsh
as my default shell, but this guide applies to bash
as well.
So, without further ado, here's how I setup my Python workspace:
Step 1. Define your requirements
Before getting into the actual configuration, you must take some time to define what your optimal setup looks like. For me, it looks like this:
- It must have Python 2.7+, Python 3.6, Python 3.7, and Python 3.8 installed.
- Python 3.8 must be the default version.
- I must be able to switch between versions easily.
- I must be able to fire up a
ipython
using the default python version I specified. - Dependency tools I use -
pipenv
andpoetry
- must work with all Python 3+ versions.
Step 2. Installing and configuring multiple Python versions
To install multiple versions and being able to switch between them, I use pyenv
.
pyenv
has several benefits such as:
- Lets you change the global Python version on a per-user basis.
- Provides support for per-project Python versions.
- Everything is installed in the
$HOME
directory. This means no risk of messing up the default Python installation. - Supports
pypy
,anaconda
,CPython
,Stackless-Python
and others!
In addition to pyenv
, I also use the pyenv-virtualenv
plugin to manage my virtual environments.
Installing pyenv
and its plugins
Since last year, I've been using Homebrew both on macOS and Linux. So, the following steps are OS agnostic for me.
brew install pyenv
brew install pyenv-virtualenv
Creating a directory for the virtualenvs
Each project should have each own virtual environment associated with. This way we can work on more than one project at a time without introducing conflicts in their dependencies. I keep all my virtualenvs under the $HOME/.ve
directory.
mkdir -p $HOME/.ve
Once the folder is created I add the path to my ~/.zshrc
(for bash users, add it to ~/.bashrc
):
cat <<"EOT" >> ~/.zshrc
# pyenv config
# Set virtualenv dir
export WORKON_HOME=~/.ve
# Initialize pyenv
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi
# Initialize pyenv-virtualenv
eval "$(pyenv virtualenv-init -)"
EOT
The variable WORKON_HOME
tells pipenv
where to place your virtual environments.
Once that's done, you can restart your shell by either closing and opening a new window or running:
exec $SHELL
Installing all Python versions I need.
PY_DEFAULT=3.8.5
PY_VERSIONS=( $PY_DEFAULT 3.7.8 3.6.11 2.7.18 )
for py_version in "${PY_VERSIONS[@]}"
do
echo -e "Installing Python $py_version...\n\n"
# Install specific Python version
pyenv install $py_version
done
Step 3. Installing the tools
For dependency management I use two different tools, pipenv
, and more recently poetry
. Eventually I will probably settle on poetry
by at the moment I need both.
Also, I rely a lot on jupyter notebooks, for quick data analysis tasks; and ipython
as a fancy Python interpreter.
Again, I don't want to pollute the global installation. To avoid that we can use pyenv-virtualenv
to create virtualenvs for the tools.
Let's install jupyter, pipenv, and poetry.
# Creating tools3 venv
pyenv virtualenv $PY_DEFAULT tools3
# Activating
pyenv activate tools3
# Upgrade pip
pip install --upgrade pip
# Install Jupyter
pip install jupyter
# Install Jupyter extensions
pip install jupyter_nbextensions_configurator rise
jupyter nbextensions_configurator enable --user
# pipenv
pip install pipenv
# poetry (using preview version to fix a bug in the 1.0.10 version)
pip install --pre poetry -U
If you use pipenv
, make sure to install its completion script. You can add it to your shell like so:
echo 'eval "$(pipenv --completion)"' >> ~/.zshrc
And for poetry
:
# Oh-My-Zsh
mkdir -p $ZSH/plugins/poetry
poetry completions zsh > $ZSH/plugins/poetry/_poetry
Since the completion is a oh-my-zsh
plugin, I need to add poetry to the plugins list in my ~/.zshrc file.
plugins(
poetry
...
)
This can be accomplished using sed
:
sed -i.bak 's/^plugins=(\(.*\)/plugins=(poetry\n \1/' ~/.zshrc
Now, we need to configure poetry
to create virtualenvs inside ~/.ve
, just like pipenv
.
echo 'export POETRY_VIRTUALENVS_PATH=$WORKON_HOME' >> ~/.zshrc
More poetry configurations
One more thing, pipenv
's default behaviour is to load any environment variables defined in the .env
in the root of the project. It does that on two occasions, when running pipenv shell
and pipenv run
. poetry
, on the other hand, does not support that. As a workaround, I need to load it manually.
To automate the process, I created a shell function that loads it whenever I run poetry shell
or poetry run
. If I want to disable it, I can set the POETRY_DONT_LOAD_ENV
variable. This is, again, similar to how PIPENV_DONT_LOAD_ENV
works.
cat <<"EOT" >> ~/.zshrc
# Get the poetry's full path
POETRY_CMD=$(which poetry)
# Allow poetry to load .env files
function poetry() {
# Define the full command. i.e. poetry [run|shell|version]
POETRY_FULL_CMD=($POETRY_CMD "$@")
# if POETRY_DONT_LOAD_ENV is *not* set, then load .env if it exists
# also, only loads when for "run" and "shell" commands.
if [[ -z "$POETRY_DONT_LOAD_ENV" && -f .env && ("$1" = "run" || "$1" = "shell") ]]; then
echo 'Loading .env environment variables…'
env $(grep -v '^#' .env | tr -d ' ' | xargs) $POETRY_FULL_CMD
else
$POETRY_FULL_CMD
fi
}
EOT
Now deactivate the virtualenv tools3
.
# Deactivating the venv
pyenv deactivate
Step 4. Setting interpreters priority
Now that we have all the versions we need installed we must establish some sort of priority. Basically, I want to use poetry
, or any other tool, without activating the virtualenv where I installed them. We can do that using pyenv global
command.
PY_DEFAULT=3.8.5
PY_VERSIONS=( $PY_DEFAULT 3.7.8 3.6.11 2.7.18 )
pyenv global $PY_VERSIONS tools3 system
Let's check:
$ pyenv versions
* system (set by /home/miguel/.pyenv/version)
* 2.7.18 (set by /home/miguel/.pyenv/version)
2.7.18/envs/tools2
* 3.6.11 (set by /home/miguel/.pyenv/version)
* 3.7.8 (set by /home/miguel/.pyenv/version)
* 3.8.5 (set by /home/miguel/.pyenv/version)
3.8.5/envs/tools3
* tools3 (set by /home/miguel/.pyenv/version)
Everything looks good, it's time to restart the shell and that's it.
Preventing accidental installation of packages
One thing that happened a lot to me was installing packages inside one of the global interpreters using pip
. As I mentioned earlier, it's a good idea to keep each Python interpreter intact. If we need to install anything, I prefer to create a new virtualenv, like I did for tools3.
Now, how can we lock each Python installation?
That's actually pretty straightforward. We can set pip
to only install packages if there's a virtualenv active.
echo 'export PIP_REQUIRE_VIRTUALENV=true' >> ~/.zshrc
What about pipenv and poetry?
Now, if I have a poetry
project that requires Python 3.6, I tell poetry
to use the 3.6 version and it will automatically create a virtualenv using the Python 3.6.11 I installed.
Example:
# pyproject.toml
...
[tool.poetry.dependencies]
python = "~3.6"
...
Output:
$ poetry env use 3.6
Creating virtualenv sandbox-aBhl6cgV-py3.6 in /home/miguel/.ve
Using virtualenv: /home/miguel/.ve/sandbox-aBhl6cgV-py3.6
$ poetry shell
Loading .env environment variables…
Spawning shell within /home/miguel/.ve/sandbox-aBhl6cgV-py3.6
$ . /home/miguel/.ve/sandbox-aBhl6cgV-py3.6/bin/activate
(sandbox-aBhl6cgV-py3.6) $
Conclusion
That’s pretty much it! I hope this tutorial is useful for you, just as it’s for me. Whenever I need to reconfigure my workspace, I follow those steps. Also, I’ll probably create a shell script to automate it instead of copying and pasting from this guide. This tutorial was initially based on https://medium.com/@henriquebastos/the-definitive-guide-to-setup-my-python-workspace-628d68552e14, which served as inspiration on how to setup my own workspace.
How I Set up My Python Workspace first appeared on miguendes's blog.
Top comments (0)