Poetry is a tool solving the problem of Python packaging. It was started back in February 2018 by Sébastien Eustace (also the author of pendulum). It has a beautiful website and a ambitious headline:
Python packaging and dependency management made easy
It has been on my radar for a while, but I never gave it a proper go. I was happily using pip-tools, which was solving my main use case, while being a lot more lightweight and meant I could keep working with pip since its output is a good old requirements.txt
. I heard about it a few times online, often next to Pipenv, but recently it looks like Poetry got a bit more traction.
Having a bit of time on my hands, a few weeks ago I decided to take a proper look at it and maybe migrate one of my projects, Deezer Python.
Starting point
Before the migration, here is how the package was managed:
- Project metadata were in
setup.cfg
, using setuptools declarative config. - Development (and documentation) dependencies managed by pip-tools.
- Documentation hosted on Read the Docs (RTD).
- Releases automated with Python Semantic Release (PSR) on Github Actions.
- Development tools configured in
setup.cfg
(black, isort, pyupgrade, flake8). - Using
tox
to help local testing. - Using Github Actions for CI.
These are a couple of features that were impacted by migrating to Poetry, and I think it’s worth mentioning them for context. Since Poetry uses pyproject.toml
, I was also hoping to move all my tools config from setup.cfg
to that file.
Migration
Installing Poetry
The first step is to get the CLI. I initially installed it via Homebrew, but later realised that Poetry was setting some default values based on the Python version its installation uses. As the Homebrew Python can be updated without notice, I realised it was not the best option here, so I later reinstalled it via pipx
using a Python installation managed by pyenv that wouldn’t be wiped without my knowledge:
pipx install \
--python ~/.pyenv/versions/3.8.6/bin/python \
poetry
Project metadata
The first step was to migrate the project metadata from setup.cfg
to pyproject.toml
. Poetry comes with a handy interactive command poetry init
which will create a minimal pyproject.toml
for you. I already noticed a few pleasant surprises:
- The CLI was very nice to interact with
- The
author
andauthor_email
fromsetup.cfg
were merge into an array ofauthors
, each with the formatFull Name <email@address.com>
.
I then went on to convert more settings into the new format manually, and the process was quite painless. Many settings have the same name and values, and when they are different, it’s mainly to simplify things. I guess it’s something a library like setuptools cannot easily afford to do due to backwards compatibility, but that a new opt-in tool like Poetry can.
Dependencies
Adding dependencies and development dependencies was pretty simple, I just needed to run poetry add [-D] ...
with the list of packages at the end.
In this process, I discovered that one of the development dependency, pyupgrade
is not compatible with Python 3.6.0: Poetry would not let me set my own Python to ^3.6
. I initially changed my minimum version to ^3.6.1
as a quick fix, but I realised later that it impacted my package's trove classifiers, the ones generated by Poetry: my package wasn't listed as Python 3.6 compatible. Since this is just for a development dependency, there is a better way! The fix is to specify the dependency conditional to the Python version:
[tool.poetry.dependencies]
python = "^3.6"
...
[tool.poetry.dev-dependencies]
...
pyupgrade = { version = "^2.7.3", python = "^3.6.1" }
With this, the minimum Python version of my package and its classifiers are correct.
Extra Dependencies
This package had an “extra require” dependency. There is a good example in the documentation, these dependencies needs to be specified in the same section as normal dependencies, but as optional
:
[tool.poetry.dependencies]
...
tornado = {version = "^6.0.4", optional = true}
And a dedicated pyproject.toml
section maps the extras to an array of optional dependencies:
[tool.poetry.extras]
tornado = ["tornado"]
I got confused initially because I thought it was done via the poetry add ... -E ...
command. However, it does something different, it’s for the extra of the dependency, not the extra of my own library the dependency should fall under.
For example Django has 2 possible extras at the moment, so one would run the following commands to use them:
poetry add Django -E argon2
It means “install Django with the argon2
extra”. I thought it meant “install Django and put it under the argon2
extra”.
Docs dependencies
The dependencies to build the docs were specified in a requirements.txt
in the docs/
folder and RTD was configured to pick this up. I initially thought that I wouldn’t be able to remove that file, but it turns out it’s possible to make it work.
Thanks to PEP 517, which Poetry is compliant with, you can do pip install .
in a Poetry package. This has been in pip since 19.0, and pip running on RTD is newer than this. However, this method wouldn’t install your development dependencies, so your docs dependencies cannot be specified as such. It works if you specify a docs
extra, though:
# pyproject.toml
[tool.poetry.dependencies]
...
myst-parser = {version = "^0.12", optional = true}
sphinx = {version = "^3", optional = true}
sphinx-autobuild = {version = "^2020.9.1", optional = true}
sphinx-rtd-theme = {version = "^0.5", optional = true}
[tool.poetry.extras]
...
docs = [
"myst-parser",
"sphinx",
"sphinx-autobuild",
"sphinx-rtd-theme",
]
# readthedocs.yml
version: 2
python:
install:
- method: pip
path: .
extra_requirements:
- docs
The downside of this, is that the extra is part of your package, so not ideal.
Releases
I recently moved the automation of releases to Python Semantic Release which worked well for me, and this would have been a blocker if it wouldn’t work. These are the pieces I needed:
- Move its config to come from
pyproject.toml
. - Package version is specified in
pyproject.toml
as well as in__init__.py
. - Update build command to use Poetry instead of setuptools.
Here is the PSR config in pyproject.toml
to achieve that:
[tool.semantic_release]
version_variable = [
"deezer/ __init__.py: __version__",
"pyproject.toml:version"
]
build_command = "pip install poetry && poetry build"
I was expecting to have to change more, but it all worked out of the box with just that.
Linting and code formatting
All the tools I use for linting and code formatting were configured via setup.cfg
and ideally I’d like to replace it by pyproject.toml
. It was possible for almost everything, except for flake8 which has an open issue for it.
I decided to move as much things as I could to pyproject.toml
, and move flake8 config to .flake8
.
With all the above, I was able to remove the setup.cfg
as well as all the pip-tools files for dependencies.
Code Coverage
Pytest had and unexpected config section for pyproject.toml
, I didn't pay much attention to it initially and put its config under [tool.pytest]
, while the section should actually be [tool.pytest.ini_options]
.
The consequence of this was that my config was ignored and I didn't run the tests with coverage enabled, which silenced another error in the coverage section tool.coverage.run
, where source should be an array:
[tool.pytest.ini_options]
addopts = "-v -Wdefault --cov=deezer"
[tool.coverage.run]
branch = true
source = ["deezer"]
With this, pytest was running with coverage, but for some reason, codecov wasn't picking up the report properly. I noticed that it used to find an xml
file when it was working, so I edited the command on CI to generate it with --cov-report=xml
:
poetry run pytest --cov-report=xml
With this changes I eventually got my code coverage reporting back, but it's something I missed in the original migration. Not directly related to Poetry, but interesting that Pytest decided to go with this section name.
Tox
Poetry works nicely with Tox, I followed the section in their FAQ, and here is a overview of the changes:
[tox]
isolated_build = true
envlist = py36,py37,py38,py39,pypy3,docs,lint,bandit
[testenv]
whitelist_externals = poetry
commands =
poetry install
poetry run pytest
...
I replaced the deps
section in each testenv
by a poetry install
into the list of commands to run, and prefixed all commands to be run in the isolated environment by poetry run
.
Github Actions
Poetry isn’t installed out of the box on Github Actions, one could either install it with a simple run
step or use a dedicated action for it. I’ve opted for the dedicated action, thinking that Dependabot could keep it up to date for me.
The rest of the changes are pretty simple, it’s a matter or replacing pip install
by poetry install -E ...
and prefixing all commands by poetry run
. My docs were tested and I was changing directory with cd
, I took this opportunity to instead use working-directory
key to the Github action step.
Verdict
Did Poetry deliver on its ambitious tagline? I think so, I was really impressed by the developer experience of Poetry, its CLI is really nice, and I hit little issues on the way. Overall the migration was not too difficult, you can check the pull request on Github. I feel like there are quite a few features I just scratched the surface (like multi-environments).
I’m going to wait a bit to see how this works in the longer run, but I think I’ll migrate my other projects soon.
Top comments (0)