A dinosaur learns poetry

Originally published on peateasea.de.

Not a real dinosaur and not real poetry. Just me updating my Python project setup knowledge.

The new¹ way,

A dino jumps:

Splash!

I’m a bit of a dinosaur. Well, I feel like one at times. I mean, I’ve been using Python since version 1.5 and that makes me feel old. Also, up until recently, I didn’t know that poetry was a big thing in the Python world for project dependency management.² The things ya learn.

Many years ago, I’d learned how to use requirements files after having read Test Driven Development in Python by Harry Percival (first edition).³ Since then, I’d simply continued using them. I’d heard of poetry over the years, but never really had any time to dig deeper. I guess that’s one of the downsides to being constantly snowed under at work. Strangely enough, now that I’ve gone freelance,⁴ I seem to have found the headspace to take more notice of such things. Maybe I just needed a change in scenery.

Over the past few months, I’ve read a few blog posts mentioning poetry,⁵ and I thought I’d check it out. Here’s what I’ve learned and how (as far as I can tell) one should set up poetry on a pre-existing project.

Please note that I’m aware that uv is the shiny new replacement for poetry that “everyone” seems to be talking about and moving to now. Let me catch up with one thing at a time, ok?

Although the poetry documentation is clear and detailed, this post is my explanation of the project setup process to myself. If it helps someone else, that’s brilliant!

Update: 2024-10-02

As it turns out, my statement about poetry being a replacement for uv isn't accurate, as pointed out by @diazona on
Mastodon. It had been my impression from what I'd read online that poetry was the replacement for requirements files and that uv was quickly becoming the "replacement" for poetry. A better way of putting this is as @diazona explains:

Poetry and uv are both among the current generation of "standard-driven" packaging tools, along with many others like hatch, flit, and pdm. Poetry happens to be among the older ones of this generation while uv is among the newer ones, but they're both doing a lot of the same stuff.

What this means is that the package management landscape in the Python world is much more diverse than I'd initially been led to believe. @diazona also pointed me to an unbiased evaluation of environment management and packaging tools, which is a thorough and in-depth discussion of the current Python packaging landscape.

Poetry prerequisite preparation

The first thing to do is install pipx. Hang on, didn’t we want to install poetry? Yeah, I thought that too when I read it. Good things come to those who wait, I guess.

pipx is a tool to install Python command line applications in isolated environments. The idea being that if you need a Python command line tool that isn’t included in your OS’s package manager, you can install it “globally”.⁶ Thus, to install poetry, first we have to install pipx.

The best way to install pipx is via your OS’s package manager, e.g. on Debian:

$ sudo apt install pipx

Because poetry is a Python command line application that we want available from anywhere within our shell environment, we install it via pipx:

$ pipx install poetry
  installed package poetry 1.8.3, installed using Python 3.9.2
  These apps are now globally available
    - poetry
⚠️ Note: '/home/cochrane/.local/bin' is not on your PATH environment variable. These apps will
    not be globally accessible until your PATH is updated. Run `pipx ensurepath` to automatically
    add it, or manually modify your PATH in your shell's config file (i.e. ~/.bashrc).
done! ✨ 🌟 ✨

Note that the poetry program won’t be available straight away in your shell (as mentioned in the warning note after the first time you’ve installed a package via pipx). Thus, you’ll need to add the pipx bin path to your $PATH environment variable so that your shell can find any programs that pipx has installed. E.g. in bash, you’ll need to do this:

$ export PATH=$PATH:$HOME/.local/bin

You’ll also need to add this path to the $PATH variable in your .bashrc so that any pipx-installed programs are available in each new shell invocation. Adding code like this to your .bashrc should do the trick:

# enable pipx bin path if available
if [-d "$HOME/.local/bin"]
then
    export PATH="$PATH:$HOME/.local/bin"
fi

poetry also comes with its own set of shell completions. The poetry team has added a handy subcommand that exports the completions for your shell. Thus, setting up the completions (in this example for bash) is as simple as:

$ poetry completions bash >> ~/.bash_completion

You’ll need to restart the shell for the completions to work.

Pre-existing project poetry

Creating a new project is easy with poetry, one only needs to run

$ poetry new <project-name>

which will create a directory called <project-name> and fill it with all the files and directory structure you will need to start a new project. For instance, using the example from the poetry documentation, here’s the structure of a project called poetry-demo:

poetry-demo
├── pyproject.toml
├── README.md
├── poetry_demo
│ └── __init__.py
└── tests
    └── __init__.py

My main use case is to replace requirements file dependency management with poetry in pre-existing projects. Thus, I’ll focus on how to handle introducing poetry to a pre-existing project in what follows.

Priming a project with poetry

To pare this example down to its bare bones, I’m only going to set up Django as a main dependency and pytest as a dev dependency. These are the two most used dependencies I have in most of my projects, so these should be sufficient to illustrate the process of migrating to poetry.

The first thing to do is to initialise a poetry configuration for the project. For this, we use the init subcommand, which asks us a few questions and creates an initial pyproject.toml configuration file:

$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [poetry-test]:
Version [0.1.0]:
Description []:
Author [Paul Cochrane <paul@peateasea.de>, n to skip]:
License []:
Compatible Python versions [^3.9]:

Would you like to define your main dependencies interactively? (yes/no)
[yes]
You can specify a package in the following forms:
  - A single name (requests): this will search for matches on PyPI
  - A name and a constraint (requests@^2.23.0)
  - A git url (git+https://github.com/python-poetry/poetry.git)
  - A git url with a revision
    (git+https://github.com/python-poetry/poetry.git#develop)
  - A file path (../my-package/my-package.whl)
  - A directory (../my-package/)
  - A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Package to add or search for (leave blank to skip): Django@4.2.16
Adding Django@4.2.16

Add a package (leave blank to skip):

Would you like to define your development dependencies interactively?
(yes/no) [yes]
Package to add or search for (leave blank to skip): pytest@8.2.2
Adding pytest@8.2.2

Add a package (leave blank to skip):

Generated file

[tool.poetry]
name = "poetry-test"
version = "0.1.0"
description = ""
authors = ["Paul Cochrane <paul@peateasea.de>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"
Django = "4.2.16"

[tool.poetry.group.dev.dependencies]
pytest = "8.2.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Do you confirm generation? (yes/no) [yes]

I used specific versions for Django and pytest above because my legacy projects can’t yet use the latest versions.

Demanding the keys to the castle

As an interesting side note, if you don’t specify a version number, you will be asked to select the package from a list and then poetry will work out the most recent version for you. For instance:

Would you like to define your development dependencies interactively? (yes/no) [yes]
Package to add or search for (leave blank to skip): pytest
Found 20 packages matching pytest
Showing the first 10 matches

Enter package # to add, or the complete package name if it is not listed []:
 [0] pytest
 [1] pytest123
 [2] 131228_pytest_1
 [3] pytest-collect-pytest-interinfo
 [4] pytest-pingguo-pytest-plugin
 [5] pygments-pytest
 [6] pylint-pytest
 [7] pytest-briefcase
 [8] pytest-buildkite
 [9] pytest-bwrap
 [10]
 > 0
Enter the version constraint to require (or leave blank to use the latest version):
Using version ^8.3.3 for pytest

Add a package (leave blank to skip):

If this is the first time you’ve done this, then a window will appear asking you to enter the password for a default keyring that it wants to create.

It’s not clear why this needs to happen at all (it would have been nice to have been warned about this in advance) and this popup window will disallow all access to any other windows and terminals, thus one can’t use Ctrl-C to exit the poetry init process. It seems that if one is sufficiently consequential in clicking the Cancel button, that it will disappear. However, the next time you want to add a package the popup will appear again.

If you decide to create a default keyring, two files will be created in ~/.local/share/keyrings/, namely:

$ ls .local/share/keyrings/
default Default.keyring

I’ve found that the presence of these keyrings can cause plain pip (in some situations) to also want to use a keyring when installing a package and hence it will present a blocking popup to request the keyring password. Unfortunately, I’ve not been able to reproduce this behaviour consistently on my test systems, so this is a warning that it might happen to you, so don’t be too surprised if such a popup appears.

To deactivate the keyring, there are a few options. One way to do this is to set the PYTHON_KEYRING_BACKEND environment variable:

$ export PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring

which then disables the Python keyring package and hence stops the keyring password popup from appearing when adding packages to your project. Note that this is a keyring specific to Python and isn't related to other keyrings which might be on your operating system.

One could also set the keyring password to a blank value when the popup window appears, thus making the keyring accessible without a password. That might not be what you want, though, so consider carefully if this is the solution you wish to choose.

There is (at the time of writing) an open issue in the pip repository about documenting this issue, so have a look there for more details and discussion.

Keep your poetry close, and your venvs closer

By default, poetry likes to keep the virtual environments that it creates within a central location.⁷ Because I like to keep my virtual environments within my project directories, I find it useful to set the virtualenvs.in-project configuration option to true. Note that this isn’t a configuration value in the pyproject.toml file; it’s a configuration option for poetry itself. Because of that, we need to use the poetry config command to set the value.

To list the current poetry configuration, use

$ poetry config --list

To set the virtualenvs.in-project config value, we use this command:

$ poetry config virtualenvs.in-project true

Now poetry will automatically create a directory called .venv within the current directory and store all the virtual environment-related stuff in there. Up until now, my personal preference has been to create the virtual environment in a directory without the leading dot, i.e.

$ python3 -m venv venv

but using the name .venv isn’t hard to adapt to. Using a dotted name is also a good idea as it hides the directory, giving less visual noise in directory listings.

To package or not to package?

By default, poetry assumes that you will want to provide any project you’re working on as a package on PyPi. To date, I’ve only been a downstream user of packages, hence this isn’t the correct setting in my situation. Thus–for me, at least–I need to set package-mode = false in the tool.poetry section of the pyproject.toml. This tells poetry that I only want to use PyPi packages in an application and makes certain parts of the pyproject.toml configuration optional.

Reconciling and receiving requirements

With the poetry and project configuration all set up, we can now install our upstream dependencies. This is as simple as running poetry install:

$ poetry install
Creating virtualenv poetry-test in /home/cochrane/.../poetry-test/.venv
Updating dependencies
Resolving dependencies... (0.7s)

Package operations: 10 installs, 0 updates, 0 removals

  - Installing typing-extensions (4.12.2)
  - Installing asgiref (3.8.1)
  - Installing exceptiongroup (1.2.2)
  - Installing iniconfig (2.0.0)
  - Installing packaging (24.1)
  - Installing pluggy (1.5.0)
  - Installing sqlparse (0.5.1)
  - Installing tomli (2.0.1)
  - Installing django (4.2.16)
  - Installing pytest (8.2.2)

Writing lock file

Running poetry install will also generate a lockfile called poetry.lock in the project’s main directory. You should add this file to version control so that builds are reproducible.

Why wouldn’t they be reproducible anyway? Well, the dependencies defined in pyproject.toml are only a subset of all resolved dependencies. Because upstream dependencies get updated often, it’s possible that–without a lockfile–someone else on your team (or the CI system) might get a different set of resolved dependencies. Such differences can cause builds to break in very confusing ways. The lockfile ensures that everyone uses the same version of all dependencies, making applications build and test reliably.

It’s nice that poetry is clear about what it’s doing when installing dependencies. This way we can see where it has created the virtual environment and thus where it has installed any project dependencies. Also, we can see clearly what packages have been installed and at what versions. The visual layout and the use of colour (if available in your terminal) help to highlight the installation details.

Becoming more dependent

Adding and installing new packages to a project is a breeze with poetry add. This avoids having to update the pyproject.toml file by hand and helps to avoid silly bugs where one gets the dependency version syntax wrong.

A well-worn workflow

Now that I’ve installed the project dependencies, I can use my tried-and-trusted workflow. In other words, I activate the venv in my shell and run pytest directly (or via a make target), for instance:

$ source .venv/bin/activate
$ pytest tests/test_basic.py
========================================== test session starts ===========================================
platform linux -- Python 3.9.2, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/cochrane/.../poetry-test
configfile: pyproject.toml
collected 1 item

tests/test_basic.py .
[100%]

=========================================== 1 passed in 0.02s ============================================

where I used a stub test file in the example here to make sure that pytest works as expected:

# -*- coding: utf-8 -*-

from unittest import TestCase

class TestBasic(TestCase):
    def test_basic(self):
        self.assertTrue(True)

# vim: expandtab shiftwidth=4 softtabstop=4

I like that it’s still possible to use this standard workflow when using poetry. Other project dependency management tools that I’ve used in the past have required that everything be done through the tool, which made being productive painful at times. It seems that poetry doesn’t impose this on its users which is a good thing in my opinion.

The rime of the ancient developer

My previous project setup workflow looked like this (equivalent to the setup discussed above):

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install Django==4.2.16
$ pip install pytest==8.2.2
$ mkdir requirements
$ pip freeze | grep -i django > requirements/base.txt
$ echo '-r base.txt' > requirements/dev.txt
$ pip freeze | grep -i pytest >> requirements/dev.txt

In this workflow, I’ve set up the virtual environment, activated it, installed dependencies and then maintained the requirements in separate files. This was a stable and reliable pattern to use when setting up a project. What I liked about this “old” way was that all required infrastructure was part of a standard Python installation. For instance, the venv module has been part of the stdlib now for a while and one only needs to use pip to install dependencies. pip freeze then comes in handy for juggling dependencies.

I think the main advantage of poetry for me is likely to be its improved dependency management. Thus, moving to a new Python version isn’t yet another trip to dependency hell. One can hope!

The “new” way does have its advantages in that all tool configuration is now centralised within the pyproject.toml file. This means any config for pylint or black or whatever is in the one location and it’s not necessary to juggle several different kinds of resource files, each using a different format. Adding and installing new dependencies is much easier with poetry add. Also, it’s nice not to have to organise separate requirements files for different purposes, such as prod and dev; these things are now a natural part of the pyproject.toml configuration.

As we all know, endings are just beginnings

Now that I seem to have the basics vaguely ok, it’s time for me to start the really hard work: updating my legacy Python projects to use poetry. I’d best hop to it!

1. Warning: your value of new may differ. ↩

2. BTW: If you google for “dinosaur poetry” you’ll find all kinds of rather sweet poems about dinosaurs for kids. ↩

3. This is a great book by the way; I highly recommend it. ↩

4. I’m available for freelance Python/Perl backend development and maintenance work. Contact me at paul@peateasea.de and let’s discuss how I can help solve your business’ hairiest problems. ↩

5. It’s somehow ironic that Python (as a language) doesn’t lend itself to writing poetry, yet a project dependency manager is named thus. Interestingly enough, another programming language I know has poems written in it and can even print error messages as haiku. Ya can’t have everything, I guess. ↩

6. This is only global in the sense that it’s available everywhere within the environment of the user that installed it. By this, I mean that any tool installed via pipx is not restricted to being available within some project directory, but is available in the user’s global $PATH. However, command line tools installed via pipx are not global in the sense that they’re available for all users on the system. ↩

7. poetry handles virtual environment venv creation for you. By default, this location is a directory with an automatically generated name (based on the project name) under ~/.cache/pypoetry/virtualenvs. ↩