In this post, I am using
PowerShell 7 with
Python 3.8.1. Adjust examples according to your environment.
If you don't care about the logic and reasoning behind "why Poetry?", skip to Getting Started where we build a simple Flask app using Poetry as our package manager.
Python has existed for almost 30 years now and many revolutions have occurred in the software arena since its inception. Paradigms have shifted around OOP and functional, design patterns have come and gone, but one issue every language sees, and has varying success in tackling, is dependency management.
For the first half of its life thus far, Python did not have a dedicated way to handle packaging or dependencies. It was the wild west: relying on what's in the standard library or figuring it out yourself. It wasn't until Python 1.6 that the
distutils package was added to the standard library in a first attempt to handle the problem in 2000. Here are some more milestones according to PyPA (Python Packaging Authority):
- 2003: PyPI was operational but wouldn't have "packages" hosted until 2005.
setuptoolsis created and is the present packaging go-to.
virtualenvpackage is created to segregate packages per-project.
- 2011: PyPA is created to take over
- 2013: PEP 425 and PEP 427 accepted which define the built-package (binary) format,
- 2016: PEP 518 accepted for the
pyproject.tomlformat for static build dependency declarations.
- 2017: PEP 517 accepted for
setuptools) build system backends.
There is a lot of history when it comes to Python packaging and decisions, but I tried to highlight the milestones. The last two are the most relevant to this post as they enabled the
Poetry project to take off as it is today and move away from the history of
When you start most Python projects, there isn't really guidance on how to start. Most guides used to give you a
requirements.txt file to define your dependencies that you install using
PS ~/> pip install --user --requirement requirements.txt
You would probably install those in your user site-packages (indicated by
--user which could cause some chaos with changing versions, conflicting packages, etc.) and off you go. This was hard to scale and deploy, however, which is why
virtualenv came along to ease that development process by keeping your dependencies contained per-project and "activating" the environment when you're using it:
PS ~/> python -m venv .venv PS ~/> .venv/Scripts/Activate.ps1
Another problem with this approach is that the requirements file is rather simple. It's typically a newline-separated plaintext file with a package name and some version indicator that a person writes or generates using:
PS ~/> pip freeze > requirements.txt
But using this approach has its own problems. It locks all the versions with exact specifiers and their dependencies:
PS ~/> pip install flask==1.1.1 # ... PS ~/> pip freeze click==7.1.1 Flask==1.1.1 itsdangerous==1.1.0 Jinja2==2.11.1 MarkupSafe==1.1.1 Werkzeug==1.0.0
This creates a lot of manual work to audit your dependencies and bring them up-to-date when needed.
A route some people go with is using
setuptools to manage their project as an installable package. This is particularly effective when your project has tests and sub-packages, but adds another layer to the stinky onion.
setuptools uses a
setup.py script to control the installation of your package. This file has one requirement when called: to execute
import setuptools setuptools.setup() # barebones
Normally this function takes arguments to imperatively install the package, but a nice, declarative way to set these arguments is using a
# ... [options] install_requires = Flask==1.1.1 # ...
This file is also used by many packages for configuration, so it's a neat way to centralize your configuration, but it has the same problems as it uses the same format for dependencies. More information on this method of setup can be found here.
This leaves us with the following problems:
- How do we isolate our project?
virtualenvis nice, but can be forgotten
- How do we manage our dependencies and their versions? Locking everything that gets installed is a management nightmare.
One last issue that deserves a little blurb: how do we install our own project/package and use it? Depending on
PYTHONPATH is fraught with complications especially when you start talking about testing your code and many a solution has been created to mangle it to fit a use-case perhaps unnecessarily.
Poetry's tagline is "Python packaging and dependency management made easy." It accomplishes this by using features introduced by PEP 517 and PEP 518 introducing independent build system backends and new dependency declaration (away from
setuptools; they're still there, but abstracted away from the user).
yarn tool. There is the
pyproject.toml similar to
package.json and a
yarn.lock. There is also configuration of the tool itself kept in
poetry.toml (again, like
Poetry is available on PyPI, but they recommend using their alternative installer to avoid polluting your Python packages.
# bash curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python # PowerShell (Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -UseBasicParsing).Content | python # python pip install --user poetry
To the meat and potatoes- using the tool! Here, the tool creates our project scaffold so the package is installable in a virtual environment:
PS ~/> poetry new --name app --src poetry-intro Created package app in poetry-intro
Now that we have a project, I go a step further to create the virtual environment local to the project for easier editor integration:
PS ~/> Set-Location -Path poetry-intro PS ~/poetry-intro/> poetry config --local virtualenvs.in-project true
With the configuration set, we're ready to initialize the environment, install our package, and add some dependencies:
PS ~/poetry-intro/> poetry install Creating virtualenv app in ~/poetry-intro/.venv Updating dependencies Resolving dependencies... Writing lock file Package operations: 11 installs, 0 updates, 0 removals - Installing pyparsing (2.4.6) - Installing six (1.14.0) - Installing atomicwrites (1.3.0) - Installing attrs (19.3.0) - Installing colorama (0.4.3) - Installing more-itertools (8.2.0) - Installing packaging (20.3) - Installing pluggy (0.13.1) - Installing py (1.8.1) - Installing wcwidth (0.1.9) - Installing pytest (5.4.1) - Installing app (0.1.0) PS ~/poetry-intro/> poetry add flask Using version ^1.1.1 for flask Updating dependencies Resolving dependencies... Writing lock file Package operations: 6 installs, 0 updates, 0 removals - Installing markupsafe (1.1.1) - Installing click (7.1.1) - Installing itsdangerous (1.1.0) - Installing jinja2 (2.11.1) - Installing werkzeug (1.0.1) - Installing flask (1.1.1)
If you're in an environment with private registries, add them according to the docs (I could not get them to work when specifying them in
[[tool.poetry.source]] name = "private-repo" url = "https://url" default = true [[tool.poetry.source]] name = "private-repo2" url = "https://url2"
default tells Poetry to ignore the public
Now we'll get to writing code:
from flask import Flask app = Flask(__name__) @app.route('/') def index(): return 'Hello, World!'
To execute our new app, we take advantage of the
run command which executes commands in the context of the virtual environment:
PS ~/poetry-intro/> $Env:FLASK_ENV = 'development' PS ~/poetry-intro/> poetry run flask run * Environment: development * Debug mode: on * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) * Restarting with stat * Debugger is active! * Debugger PIN: 000-000-000
If you want to work in your environment for an extended period, you may want to use
poetry shell to spawn a new shell according to
$SHELL with the environment activated.
I usually have some linters and formatters to integrate into intellij/vscode for consistency, so let's add them here as developer dependencies:
PS ~/poetry-intro/> poetry add --dev yapf flake8 Using version ^0.29.0 for yapf Using version ^3.7.9 for flake8 Updating dependencies Resolving dependencies... Writing lock file Package operations: 6 installs, 0 updates, 0 removals - Installing entrypoints (0.3) - Installing mccabe (0.6.1) - Installing pycodestyle (2.5.0) - Installing pyflakes (2.1.1) - Installing flake8 (3.7.9) - Installing yapf (0.29.0)
You may notice a lack of test runner, but that was already included in the
pyproject.toml dependencies when using
poetry new. Speaking of which, no project is complete without some tests:
from app import app app.testing = True client = app.test_client() def test_index(): response = client.get('/') assert b"Hello, World!" in response.data
PS ~/poetry-intro/> poetry run pytest ======================== test session starts ======================== platform win32 -- Python 3.8.1, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 rootdir: ~/poetry-intro collected 1 item tests\test_app.py . [100%] ========================= 1 passed in 0.30s =========================
This post is starting to get a bit long, but I feel like I've covered the bases to get started. There is a lot more to Poetry, such as building packages, deployment to registries, etc., but the docs are a great location to dig deeper.
In our sample project, we:
- showcased using Poetry for all interactions with the project which made forgetting our virtual environment an impossibility
- saw the lockfile in action which enables repeatable builds and avoids dependency nightmares
- avoided the overhead of
setup.pyto install our package/module in our environment
I hope this project grows into further maturity and sees more adoption because it could be the catalyst to make
pip better and/or become the standard like