All you need to know about python virtual environments

Almost everyone who has worked on any project and used python packages have had to deal with different versions of packages across different branches and projects. Virtual Environments make this process easier.

Why shouldn’t I install python packages directly on my system?

Well, you can, but you have to deal with it in the future. As more and more programs and projects today get written in python, you cannot expect all of them to use the same packages of same versions. Sometimes testing branch of your project may be using an updated version of the library while the production may use an old tested one. Your system dependencies based on python (certainly in case of Ubuntu ) could break if you decide to play with python packages on your system and nobody wants that. One way or the other, it’s better to know virtual environments in detail.

Installation and Usage

We will use virtualenv to manage a separate virtual environment in this tutorial.

1. Start by creating a simple test directory or a project directory. Move into that directory using your command line.

2. Install virtualenv in your system using the command pip install virtualenv or pip3 install virtualenv in case you are using Python 3.

Note: Python has included venv module in the standard library 3.3 onwards.

1. Type the command virtualenv venv to create a virtual environment by the name of venv, this will create a folder named venv which will have all the python executables we need to start working on a separate environment for the project without disturbing the existing system setup and dependencies of other projects. This will give you the standard working environment and uses your default system python interpreter as the language for the environment by default.

2. Example: The syntax is virtualenv name_of_virtual_env_folder where name_of_virtual_env_folder can have any name without spaces. You can also omit the name by just typing virtualenv, but the files will be created in your current project directory.

3. You can also change the python interpreter you are using by running the command virtualenv -p /usr/bin/python3.6 venv . which will give you the python 3.6 (replace 3.6 with any valid version that you like, which is installed on your system) inside your virtual environment.

4. You can add venv to your .gitignore if you don't want it to get written to your source control. (It will be ignored by git)

To start the virtual environment, you can type. source venv/bin/activate Please remember to run this every time you want to work on the project or you can add it to your setup script When the virtual environment is activated your shell prompt will look something like (name_of_virtual_environment) yourusername@yourcomputername:~/project_directory$.

As always, the way to activate the virtual environments on Windows is slightly different. You can do that as mentioned in Hitchhikers Python guide as C:\Users\SomeUser\project_folder> venv\Scripts\activate.

From now on, any package that you install using pip will be placed in the venv folder, isolated from the global Python installation on your. You can keep your environments consistent by "freezing" all the packages that you're currently have in your virtual environment in a file called requirements.txt. This enables other developers to maintain consistency across projects and branches in projects.

To do that run pip freeze > requirements.txt you can later use this in new environments to install the same packages again if required by typing pip install -r requirements.txt.

When done working with the project, don’t forget to deactivate by typing source deactivate and you will be back to your system default interpreter. You can also delete the virtual environment by just removing the folder.

virtualenvwrapper

virtualenvwrapper provides a set of commands which makes working with virtual environments easier and places all the virtualenv's in one place. You can view the install instructions in the documentation and follow the procedure for your OS and your shell. (BASH, ZSH, etc.).

Some basic examples of using virtualenvwrapper

You can install and setup the wrapper using the following commands.

pip install virtualenvwrapper
export WORKON_HOME=~/Envs
source /usr/local/bin/virtualenvwrapper.sh

when you execute the export WORKON_HOME=~/Envs command, it will place all the future virtual environments created using virtualenv wrappers in ~/Envs folder.

Creating and working with a project

As illustrated in the documentation, it is quite easy to create a project and maintain it using the wrapper.

1. Create a virtual environment: $ mkvirtualenv project_folder

This creates the project_folder folder inside ~/Envs.

1. Work on a virtual environment: $ workon project_folder

Alternatively, you can make a project, which creates the virtual environment, and also a project directory inside $WORKON_HOME, which is cd-ed into when you workonproject_folder.
$ mkproject project_folder

virtualenvwrapper provides tab-completion on environment names. It helps when you have many environments and have trouble remembering their names.

workon also deactivates whatever environment you are currently in, so you can quickly switch between environments.

1. Deactivating is still the same:
   $ deactivate

2. To delete:
   $ rmvirtualenv venv

For more commands and details, you can see the virtualenvwrapper documentation.

Advanced usage and Pipenv

Virtualenv gives you the low-level visualization feature with basic environmental isolation. However it lacks others features of a full-fledged package manager, That is where the Pipenv comes in. It has a host of features that can create and manage packages for complex projects at scale. I will go into it in brief as the documentation for pipenv is pretty easy to read and well written.

Installation and Basic usage

Pipenv stores all the package information in a file called Pipfile and Pipfile.lock which has all the details of packages, versions, and source(Pypi or custom Github source).

A simple example of Pipfile and Pipfile.lock is given below (Edited version from the documentation)

Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
​
[packages]
numpy = "*"
​
[dev-packages]
pytest = "*"

Pipfile.lock: (My Pipfile.lock after creating a fresh Pipenv environment and installing numpy)

{
    "_meta": {
        "hash": {
            "sha256": "532f630f67db39ae5920f79895733204f9869909ded64df233d99611b657a39c"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "2.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "numpy": {
            "hashes": [
                "sha256:1980f8d84548d74921685f68096911585fee393975f53797614b34d4f409b6da",
                "sha256:22752cd809272671b273bb86df0f505f505a12368a3a5fc0aa811c7ece4dfd5c",
                "sha256:23cc40313036cffd5d1873ef3ce2e949bdee0646c5d6f375bf7ee4f368db2511",
                ....
            ],
            "index": "pypi",
            "version": "==1.16.2"
        }
    },
    "develop": {}
}

I will explain in a minute what these things mean and how to start working with Pipenv First, let's go ahead and install the package

Type pip install pipenv to install the package.

Make a project folder or navigate into the project folder.

$ mkdir env_experiments
$ cd env_experiments

You can now use pipenv install to setup an existing environment using a pipfile or it creates a new pipfile for you. If you only have a requirements.txt file available in the same folder when running pipenv install, pipenv will automatically import the contents of requirements.txt file.

If you want to install a package for your existing project, you can go ahead and type pipenv install <package_name> without the <> brackets. This will create a Pipfile if it doesn't exist. If it does exist, it automatically gets edited with the new package you provided.

Now you can type pipenv shell and vol-la! You are in your virtual environment.

You can type python --version and find out what version of python is your virtual environment running.

Installing and running packages

You can install any package by running pipenv install <package_name> inside the virtual environment, for version-specific installations of packages, use the ~=.

$ pipenv install requests~=1.2 # equivalent to requests~=1.2.0

This will update your Pipfile to reflect this requirement, automatically.

You can also make version comparisons using <= => . An example is given below

$ pipenv install "requests>=1.4"   # will install a version equal or larger than 1.4.0
$ pipenv install "requests<=2.13"  # will install a version equal or lower than 2.13.0
$ pipenv install "requests>2.19"   # will install 2.19.1 but not 2.19.0

Upgrading packages

This is simple and straightforward. You can run pipenv update. for updating at the packages in the project or for package specific updates run pipenv update <package_name> You can also check if there are any updates using the command pipenv update --outdated and get rid of that package.

Specifying Python version

You can specify the python version that the virtual environment will use by running $ pipenv --python version_number

Example:

$ pipenv --python 3 # Creates Python3 environment
$ pipenv --python 3.6 # Createst Python 3.6 environments

When given a Python version, like this, Pipenv will automatically scan your system for a Python that matches that given version.

If a Pipfile hasn't been created yet, one will be created for you, otherwise, the python version information will be appended at the end of the existing Pipfile that looks like this:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
​
[dev-packages]
​
[packages]
​
[requires]
python_version = "3.6"

If you don’t specify a Python version on the command–line it falls back to whatever your system’s default python installation is, at time of execution.

Managing virtual environment

$ pipenv install

$ pipenv install is used for installing packages into the pipenv virtual environment and updating your Pipfile.

Along with the basic install command, which takes the form:

$ pipenv install [package names]

$ pipenv uninstall

$ pipenv uninstall supports all of the parameters in pipenv install, as well as two additional options, --all and --all-dev.

$ pipenv lock

$ pipenv lock is used to create a Pipfile.lock, which declares all dependencies (and sub-dependencies) of your project, their latest available versions, and the current hashes for the downloaded files. This ensures repeatable, and most importantly deterministic, build.

That is more than enough to get you going with Virtual environments and Python. Many credits go to Kenneth ☤ Reitz for creating such beautiful library and writing excellent documentation for it, I have taken most of the code snippets for Pipenv examples and some explanations from the documentation.

You can read more of my articles and follow me here on dev.to or Medium or Twitter.

Comments and suggestions welcome.

References and further reading

1. Virtual environments on Hitchhikers Guide to Python
2. Pipenv Documentation: The Basics
3. Pipenv Documentation : Advanced
4. FAQ on Pipenv