This article started as a random thought and then this post on Mastodon.
It feels like Python packaging has changed pretty substantially over the past few years. When I started creating Python packages the preferred approach was with
setup.py and (what felt like) a myriad of other random files that were needed like
setup.cfg, etc. I basically found a guide once and then copied and pasted files around once I seemed to get something working.
I was an early adopter of
pipenv with the primary appeal being lock files which promised consistent deployments. My early excitement got me burned multiple times and I looked around some more.
Poetry being the cool, new upstart at the time. It completely eschewed
requirements.txt, and other disparate files -- it had one
pyproject.toml file to configure a project, developer-friendly tooling to add dependencies, a lock file for consistent deployments, and an algorithm to make sure that a tree of dependencies was all compatible with each other (which was not possible with that version of
pip at that time).
I was in love. There were definite rough patches and there were a few bugs that caught me here or there, but overall
Poetry has been my go-to for a while.
However, in the past few years,
Python packaging seems to be going through a new era of innovation.
pip released a dependency resolver in 2020 that now complains when dependencies conflict. Also in 2020, PEP 621 and PEP 631 standardized
pyproject.toml as the new normal for Python packages instead of
setup.py. PEP 517 and PEP 660 created standards for Python build systems.
Based on those PEPs,
Poetry isn't the only forward-thinking package manager anymore.
pipenv is still around (and I presume more stable at this point!), but
Rye all promise a pleasant developer experience when creating Python packages.
Most of these package managers are of the "all in one" mentality and provide similar benefits: adding dependencies from the command-line, lock files, building, and publishing packages. If you want to do this manually Packaging Python Projects lays out the steps necessary. Reducing the number of tools required to create new Python packages means there is only one place to look for documentation and keep up to date.
My first modern Python package manager. I would say the documentation is still the best available, although it might be too "designed" for some. It does a lot -- maybe too much? -- but, in my opinion, it pioneered a lot of features that are now expected in other Python package managers. I get the feeling it sometimes pushes forward without waiting for official PEPs, although I think that has changed over the past few years.
Occasionally lock files are not backwards-compatible which can be challenging when working in a team -- everyone needs to update their
Poetry install at the same time.
Another annoying feature lacking in
Poetry that is available in every other option is the ability to define "scripts". For example,
poetry run dev as an alias for
poetry run manage.py runserver 0:8000. That is not available in standard
Poetry, although I use
Poetry plugin) to provide that functionality.
Hatch never deviates from the Python PEP standards and brings a few innovative features to the table, including being able to group dependencies and scripts into custom environments. The
hatch new [project] command is surprisingly opinionated and has settings for
ruff (my new favorite Python tool),
coverage. I'm here for all of the tool choices, but I would be lying if I did not say it was a little surprising.
Under the PyPA GitHub organization which probably gives it more of a sheen of officialdom than other tools.
PDM for some reason feels like the sleeper option compared to the rest. I have not used it too much other than to set up a new project and play around with it a little bit. Dan Sloan chimed in on Mastodon pointed out that
PDM follows all of the current PEP standards, has good support for cross-platform lock files, and also includes a useful
pdm export command to output a standard
rye seems to have had a quick and meteoric rise. It was developed by the creator of Flask, Armin Ronacher, based on his opinionated approach to building
Python packages. It reminds me of
black a little bit in which a respected developer made an opinionated tool and it caught fire in the community.
Rye also provides an overview of its philosophy here.
Rye has multiple cautions in the docs that it is experimental and not production-ready, although I did not see any glaring issues with my simple testing.
Rye is a little different than the other options because it mostly consolidates a set of other tools (
pip-tools, etc.) into a particular workflow. Another distinguishing factor is that it handles multiple installing
Python versions directly --
Poetry recommends using
pyenv to deal with the same use case.
I do not believe GitHub stats are especially useful, but the number of stars do give some indication of the community engagement with a particular tool (current as of 2023-11-12).
Personally I have multiple libraries using
Poetry, one using
Hatch, and now one using
Rye. The next one might use
PDM. 😎 I like trying new tools to see how they work and to improve my workflow.
One could look at all of the options for Python packaging and be frustrated by the paradox of choice. I understand that viewpoint, but I also appreciate that all of these tools are making different trade-offs and pushing the state of the art forward in their own ways. Maybe at some point in the future there will be one "approved" Python package manager ala
Cargo, but I am doubtful. There are pros and cons with multiple options, but I tend to think it's useful overall.
I personally have not used each package manager extensively. But, I have started a new project from scratch with each, did some simple tasks, and read through all of the documentation. Each project is relatively mature, so you can just choose any of the options and you would be fine. Standardizing on
pyproject.toml means that the conversion from one package manager to another is usually just updating some
TOML. So, choose the package manager that speaks to you (or try them all!) and improve your developer experience henceforth. 🛠️