Level up your pip install skills

In this article we are looking at different ways to install code into your virtual environment with pip.

These will increase in complexity, but fret not, I am there for you every step of the way. pats your back

Enough talk! Let's start with something easy.

Installing a local repository

Assume the following situation: You just checked out a repository and want to install the requirements.

This can easily be done by using the following command ... of course after you have created a virtual environment:

$ python -m venv (name of virtual environment)
$ source (name of virtual environment)/bin/activate
$ pip install .

If you are wondering about the install command and the absence of a requirements.txt, I have bad news for you. It is 2024 and you should no longer use a requirements.txt.

This is of course only my own opinion, but all repositories I am working with have a pyproject.toml and I would strongly recommend to use one in every of your projects as well. Viable exception might be sandbox projects and small scripts.

The "why" would be misplaced here, but allow me to give you a sneak peek. Not only does it allow you to define your requirements themselves. You can also define optional dependencies the user can install if needed.

This is especially useful for development tools, that you do not want in the productive application, like testing libraries or formatters.

But this is only the beginning of the list of features. They are also a place for metadata and allow for custom entry points to your application.

Here the install command again:

$ pip install .

Make sure that you are in the folder where the pyproject.toml can be found.
Here a pro-tip, use the following command if you intend to work on the repository you are installing

$ pip install -e .

This performs an editable install also known as "Development Mode" which allows you to iteratively test your code changes without the need to reinstall your project.

What does that mean?

Did it ever happen to you that you changed code in a module you are importing from, but the changes do not seem to take effect?

Try editable installs!

Interlude: Installing from a branch

Quick question: What do you do, if you want to install the version of a specific branch instead of the default branch?

The answer is obvious

$ git checkout (branch name)

and repeat the steps above! Right?

You fool, you just activated my trap card!

See, ever since version 2.23 there is a new kid in town which allows for more intuitive switching of branches and its name is git switch.

git checkout has been deprecated ever since.

So do not expose yourself in front of you colleagues by using out-dated tools. Instead, casually drop a git switch next time you are sharing your screen to let everyone know that you mean business.

Install from a private repository

Brace yourself!

Everything up to this point was mere child's play. Now it is time for some big boy pip usage.

See, everyone can install packages available in a package repository, but only knowing how to install from there will mean that all the gold in private repositories will remain inaccessible to you.

It is also helpful to test your own code before turning it into a package.

If you ever find yourself in such a situation, use this command:

$ pip install git+ssh://git@(your provider)/(owner)/(repo name).git

Here an example without the placeholders, that might make it easier to understand.

$ pip install git+ssh://git@github.com/pandas-dev/pandas.git

Fun fact: everything after the '://' is almost identical to the ssh command generated by git. But notice, instead of the colon used to separate 'github.com' and the owner 'pandas-dev' a slash has to be replaced.

What if you want to install from a branch ... or any other reference for that matter?

Easy! Just add a @(ref) at the end of the command. So it can look something like

$ pip install git+ssh://git@github.com/pandas-dev/pandas.git@1.5.x (branch)
$ pip install git+ssh://git@github.com/pandas-dev/pandas.git@v2.2.2 (tag)

Private Repos and the pyproject.toml

But what if it is not enough to install packages from the command line? What if also your build pipeline should install from a private repository?

Hopefully, you agree that adding individual pip install statements to your pipeline is out of the question.

So instead, let me show you what to add to the dependencies section of the pyproject.toml. You will see, that it is very similar to the previous command:

"pandas@git+ssh://git@github.com/pandas-dev/pandas.git@1.5.x",

With this added, run again pip install -e ..

Congrats! You have just installed an outdated version of pandas into your environment. You might want to repeat that with the actual package you need.

Now make it fast

Since you stuck with me until this point, I will throw in a bonus tool recommendation.

These past couple of months I have used uv, which is a drop-in replacement for pip (among other often used tools in the python eco system) written in Rust.

The biggest selling point is that it significantly speeds up the creation of virtual environments and the installation of packages. Especially if you are recreating virtual environments, because it uses caching. We are talking about being 10 times faster ... or even 100 times faster if the cache is warm.

The list of benefits is much longer than that, but also this is a thing for another article. So, give it a try for now and thank me later.

Conclusion

Let's wrap this thing up.

These were all the ways I have used during my work when it comes to the interplay between git and pip. There might be other ways to install stuff, but this should cover 99% of use-cases.

Have I forgotten your favorite command line trick? Then share it in the comments.

I hope you learned something new with this article and if you are interested in more technical articles about Software Development, then consider a follow.