If there's something I've learnt with experience is that there's almost always a simple way to do things. There are millions of people dealing with development problems the same way you are, so chances are someone already stumble with a way to simplify things.
One of those simple things is putting together a Python development environments.
I'm wouldn't say I'm a Python developer. I only use it for a specific subset of cases, mostly requiring large amounts of data for analysis, Machine Learning, etc.
So intuitively, when setting up my development environment I try to make it as easy as possible, and coming from other programming languages and frameworks such as Ruby, Elixir, Node, etc where there's clear separation between version language and libraries almost by default, I'd expect Python to be no different.
Python's setup is a mess. Python 2 is still around and it has no compatibility with Python 3. Python 3 at the same time has so many ways to be setup that it makes it so confusing.
If you pick the wrong way to setup and tie too many things to the system, then you end up with incompatibilities across projects and all other sorts of issues.
However, there's one type of setup that, in my opinion, seems to be the most similar to what I'm used to. And it's using Pyenv along with Pipenv to setup projects individually and isolated within their own Python and library versions.
Let's go ahead and dig deeper into this.
Pyenv is a Python version manager tool. It was based on the popular Ruby version managers
Its functionality is quite simple. It lets you have different versions of Python in your environment, which are not dependent on each other or the system.
There are many other tools that sort of do something similar, but I find Pyenv to be the most unintrusive of all. It does not require Python itself as it's built with shell scripts, and it does not need to be loaded into your shell.
One additional thing to remember is that you don't want your system Python running the show. So the more you can stay away from it and use isolated environments the better.
There are many ways to install Pyenv.
By far the easiest if you're on MacOS or Linux is via Homebrew
❯ brew install pyenv
After this, you need to let your bash initiate pyenv on start. For this you can add the following init script to whatever shell you're using.
In my case for
zsh I need to add it to my
.zshrc but for you it could be
if command -v pyenv >/dev/null 2>&1; then eval "$(pyenv init -)" fi
This will allow Pyenv to manipulate your path and give itself priority inside pyenv environments.
And it's done!
For more installation instructions you can check their Github page.
Next thing to do is install a new version of Python. You can install as many as you need, even from different distributions.
To see what versions are available, use
❯ pyenv install --list
This will give you a full list of available versions.
Let's go ahead and install Python 3.9.1
❯ pyenv install 3.9.1
Just like that, you're rolling on Python 3.9.1, yay!
Last step is to tell your system the global version you want to use. You don't necessarily have to do this, but it makes things a bit easier down the line
❯ pyenv global 3.9.1
Congratulations, you have Python in your system 🙂
Pipenv works at a lower level than Pyenv, and they can work together very well.
Pipenv is a tool, again sort of inspired in Bundler, Npm, Yarn etc, that aims to keep your project environment nice and tidy.
It might be counter-intuitive and you may be tempted to just install everything at the global scale so it's available for every project you use, but don't! This is an easy way to depend on libraries that may not be available when using somewhere else.
Additionally Pipenv makes collaboration really easy. Stakeholders all use the environment setup in the Pipfile and this guarantees everyone is on the same page.
So what does Pipenv do exactly?
It uses a
Pipfile.lock to sort of snapshot the dependencies you use in a project. It allows you to work in an enclosed environment, without the fear of your work slipping out to your other project environments.
Once you have Pyenv running and are on your preferred version of Python, run
❯ pip install -U pipenv
Pyenv sort of hijacks the
pip command, to make sure you're working in the correct environment.
After this you're ready to start using
The library is meant to be used within a project or project folder/context, so go ahead and navigate to your preferred project directory and initialise pipenv in there
❯ pipenv install
This will create a
Pipfile containing your dependencies, and a
Pipfile.lock containing the versions and dependencies of those dependencies.
Traditionally, another way to manage dependencies in Python projects is using a
requirements.txt file. If you already have one of those on an existing project, pipenv will transform it to a
Pipfile with the previous step and attempt to migrate your existing dependencies to it.
From now on, you're good to start using pipenv to manage your dependencies.
To add a new one you can let pipenv know what you want to install and it will update both the
Pipfile and the
Pipfile.lock for you
# Installing Jupyter notebooks ❯ pipenv install notebook
Just like that your new dependency has been added your your
Now to "enter" the environment that pipenv has created for you, you need to activate a new shell using
❯ pipenv shell Launching subshell in virtual environment…
This will load your current bash with the dependencies installed within your pipenv environment. Every command that you run within this shell will execute in the context of your Pipenv.lock.
To exit the shell just type
There's so much to learn about Pyenv and Pipenv, that you're better off just following the documentation and trying it yourself, but I guarantee you it will save you countless hours of configuration and figuring out what's going on with your project dependencies and why some projects are failing after updating other unrelated ones.
If you want to know more about the different environment setups that Python has to offer and its differences, there's this really detailed StackOverflow post that explains it very well. Even though their recommendation is using
pip instead, it's worth having a look at the different options summarised so you can make an informed decision of what you want.