Python Package Management Made Easy: A Comprehensive Guide to Poetry and pip

Simplify your Python projects with efficient package management: A comparison of Poetry and pip.

Introduction:

Managing dependencies and package installations is a crucial aspect of Python development. In this article, we will compare two popular package management approaches: Poetry and pip. We’ll explore their similarities, differences, and provide practical examples to help you make an informed decision for your Python projects.

Poetry: Poetry takes a modern approach to package management and offers a streamlined workflow for managing dependencies.
pip: Pip is a traditional approach to package management and is widely used in Python projects.

Setting Up:

Before diving into the comparison, let’s cover the first-time setup process for both Poetry and pip.

To set up Poetry:

• Install Poetry on your local machine using the official installation instructions for your operating system.

• Navigate to your project directory and run poetry init to initialize a new Poetry project.

• Follow the prompts to provide project details such as name, version, and dependencies.

• Once initialized, run poetry add package_name to add dependencies to your project.

To set up pip:

• Ensure you have Python and pip installed on your machine.

• Navigate to your project directory and create a requirements.txt file.

• List the project dependencies in the requirements.txt file, each on a separate line, in the format package_name==version.

• Run pip install -r requirements.txt to install the project dependencies.

Similarities:

Both Poetry and pip aim to simplify package management in Python projects. They share the following similarities:

1. Package Installation:

Both tools allow you to install packages and their dependencies effortlessly. For Poetry, use the command poetry add package_name, and for pip, use pip install package_name.

2. Dependency Management:

Both tools handle dependency resolution and installation. They analyze the specified dependencies and install the required packages.

Differences:

While both Poetry and pip serve the same purpose, they have distinct approaches and features. Let’s explore their key differences with examples:

1. Dependency Specification:

Poetry: Poetry embraces a declarative approach to dependency specification. In Poetry, dependencies are defined in the pyproject.toml file. You can simply run poetry add package_name to add a package, and Poetry automatically updates the pyproject.toml file with the dependency information.

# Adding a dependency
$ poetry add requests
# pyproject.toml is automatically updated with the dependency information

pip: In this approach, you manually specify the dependencies in the requirements.txt file using the package_name==version format. You have to manually update the file whenever you add or remove a dependency.

# Adding a dependency
$ echo "requests==2.26.0" >> requirements.txt
# Manually update requirements.txt with the dependency information

2. Dependency Resolution:

Poetry: Poetry performs advanced dependency resolution to ensure compatibility and consistency among packages. It creates a lock file (poetry.lock) that guarantees reproducible installations. The lock file captures the exact versions of all installed packages and their dependencies.

# Install packages with resolved dependencies
$ poetry install

pip: Dependency resolution with pip is less sophisticated. It installs the specified package versions and their direct dependencies but does not generate a lock file. This can lead to potential issues if different versions of the same package are required in different environments.

# Install packages without lock file
$ pip install -r requirements.txt

3. Environment Management:

Poetry: Poetry provides built-in virtual environment management. It creates isolated virtual environments for each project, ensuring package isolation and preventing conflicts between projects. Poetry simplifies environment management by automatically handling the creation and activation of virtual environments.

# Create and activate the virtual environment
$ poetry shell

pip: While pip itself does not offer virtual environment management, you can use tools like venv, conda alongside requirements.txt to manage virtual environments. This requires additional manual steps to set up and activate the environments.

# Create a virtual environment
$ python -m venv myenv

# Activate the virtual environment (based on your OS)
$ source myenv/bin/activate # Linux/macOS
$ myenv\Scripts\activate # Windows

4. Dev Dependencies:

Poetry: Poetry allows you to specify dev dependencies separately, which are the dependencies required for development and testing purposes only. These dependencies are not installed in the production environment.

4.1 To add a dev dependency in Poetry:

Poetry:

• Run poetry add package_name --group dev to add the package as a dev dependency.

• Dev dependencies are specified in the pyproject.toml file under the [tool.poerty.dev-dependencies] section.
  

    #Adding a dev dependency
    $ poetry add pytest --group dev
    #pyproject.toml is automatically updated with the dev dependency information

    [tool.poetry.group.dev.dependencies]
    pytest = "^7.2.0"

We can make this dev dependencies optional by using the following config in pyproject.toml file:

[tool.poetry.group.dev.dependencies]
    optional = true
    pytest = "^7.2.0"

Optional groups can be installed in addition to the default dependencies by using the --with option of the install command

    poetry install --with dev

pip: It does not have a built-in mechanism for dev dependencies. You would typically include all dependencies, including dev dependencies, in the requirements.txt file.

In requirements.txt, you can include dev dependencies alongside regular dependencies by separating them into different sections. Here’s how you can add dev dependencies to requirements.txt:

1. Create a requirements-dev.txt file:
Start by creating a separate file called requirements-dev.txt (or any name you prefer) to list your dev dependencies.

2. Add dev dependencies to requirements-dev.txt:
In the requirements-dev.txt file, list your dev dependencies using the same format as regular dependencies ( package_name==version), each on a separate line.

Example (requirements-dev.txt):

pytest==6.2.4
coverage==5.5

3. Include requirements-dev.txt in requirements.txt:
In your main requirements.txt file, add a reference to requirements-dev.txt using the -r flag and the path to the dev requirements file.

# Regular dependencies
requests==2.26.0
beautifulsoup4==4.10.0
# Dev dependencies
-r requirements-dev.txt

4. Install dev dependencies:
To install the dev dependencies, use the -r flag with pip and provide the path to requirements-dev.txt.

Example (pip command):

pip install -r requirements-dev.txt

By following this approach, you can separate your regular dependencies and dev dependencies, keeping them organized and allowing for easier management of your project’s dependencies.

Refer this StackOverflow question for more details.

5. Viewing Outdated Libraries and Updating Packages:

Poetry provides convenient commands to view outdated libraries and update packages in your project:

1.Viewing Outdated Libraries:

Poetry:

# View outdated libraries
$ poetry show --outdated

pip:

# Use a third-party tool to view outdated libraries
$ pip list --outdated

2. Updating Packages:

Poetry:

# Update all packages to their latest compatible versions
$ poetry update

pip:

    # Manually edit the requirements.txt file to specify updated versions
    # Then run the pip install command to update the packages
    $ pip install -r requirements.txt

Notable Poetry Capabilities:

1. Loading Environment Variables:

Poetry offers a way to load environment variables from a .env file. By adding a .env file in your project directory, you can define environment variables and have them automatically loaded when running commands through Poetry.

To load environment variables with Poetry:

1. Create a .env file in your project directory.
2. Define environment variables in the .env file, such as API_KEY=your_api_key.
3. Poetry will automatically load the environment variables when executing commands.

2. Custom Scripts:

Poetry allows you to define custom scripts in the pyproject.toml file. These scripts can be used to execute custom commands or perform specific tasks related to your project.

To define a custom script with Poetry:

1. Open the pyproject.toml file.

2. Add a [tool.poetry.scripts] section and define your custom script(s) as key-value pairs.

   [tool.poetry.scripts]
   test = "pytest tests/"

You can then execute the custom script by running poetry run script_name (e.g., poetry run test).

3. Python Version Management:

Both Poetry and pip support switching between different Python versions. However, Poetry provides a more streamlined approach.

To switch Python versions with Poetry:

1. Ensure the desired Python version is installed on your machine.
2. Specify the Python version in the pyproject.toml file under the [tool.poetry.dependencies] section.
3. Run poetry install to switch to the specified Python version.

Conclusion:

Both Poetry and pip offer solutions for managing Python packages, but Poetry provides a more modern, comprehensive, and developer-friendly approach. It simplifies package management by automating tasks such as dependency resolution, virtual environment management, and script execution. The built-in features of Poetry, such as dev dependencies, environment variable loading, and package update commands, enhance productivity and ease the development process.

While pip can still be suitable for python projects or those preferring a more traditional approach, Poetry emerges as the recommended choice for complex dependency management, streamlined installations, and an improved developer experience.

Consider your project’s requirements, complexity, and personal preferences when choosing between Poetry and pip . Select the tool that aligns best with your workflow, simplifies your development process, and promotes efficient package management.

By leveraging the power of Poetry, you can enhance your Python projects’ stability, reproducibility, and maintainability, making package management a breeze.

This was originally posted in medium

Ragunath Rajasekaran is a Senior Software Lead Engineer at Optum, with extensive experience as a polyglot developer. He is proficient in SpringBoot, Big Data technologies (Spark and Hive), React, and iOS development. He specializes in AWS cloud platform, employing infrastructure as code (IAC) with Terraform. Ragunath actively engages on platforms such as LinkedIn, GitHub, Dev.to, Medium.com, Medium through email.

If you haven’t already, you may want to check out my previous articles on SpringBoot, Docker, the VSCode development container, Python, React & AWS