DEV Community

Takashi Narikawa
Takashi Narikawa

Posted on • Updated on

My Python Project Development Guide ver.202305

What is this

  • The guide to set up a Python project for production deployment.

  • This guide assumes you are using Poetry for package management and adheres to the rules specified.

  • If you are not familiar with how to use Poetry, please read the following documentation and install it locally:

  • By following this guide, you can set up a Python project for production deployment using Poetry as the package manager, enforce code quality with linters and formatters, automate checks with GitHub Actions, configure your IDE to maintain consistency, and write comprehensive unit tests to ensure the project's reliability throughout the development process.

  • We recommend using Python version 3.9 or higher for your project.

  • The majority of this document was created using ChatGPT-4.

  • Sample Project


1. Project Setup/ Set up Local Environment with Poetry and Pyenv

  • Create a project directory and navigate to it:
mkdir ${project_path}
cd ${project_path}
Enter fullscreen mode Exit fullscreen mode
  • Use pyenv to manage local Python versions and ensure a consistent environment for all team members. To do this, include a .python-version file in the project.

  • Set the desired Python version for the project using pyenv:

pyenv install ${python_version}
pyenv local ${python_version}
Enter fullscreen mode Exit fullscreen mode
  • Replace ${python_version} with the desired version, e.g., 3.9.7. This command will create a .python-version file in the project directory.

  • Ensure that all team members use the same Python version specified in the .python-version file.

  • Copy the provided pyproject.tool file to the project directory and install dependencies:

  • Copy pyproject.yml example and arrange by yourself

    • For the local environment, use Poetry to create a .venv virtual environment in the project root. Place the following poetry.toml file in the project directory:
path = ".venv"
in-project = true
Enter fullscreen mode Exit fullscreen mode
  • Execute the following commands to set up the virtual environment and install the required packages
poetry env use ${python_version}
poetry install

// Run command in the created environment.
poetry run <YOUR_COMMAND>
// Enter the created environment.
poetry shell
Enter fullscreen mode Exit fullscreen mode

In the pyproject.toml file, we configure various modules for linting, formatting, and task running. Here's an explanation of these modules:

  • Taskipy

Taskipy is a task runner that allows you to automate various tasks in your Python project. In this project, we use Taskipy to define and run tasks such as formatting, linting, and testing. The tasks are configured in the pyproject.toml file under [tool.taskipy.tasks].

For more information about Taskipy, check out the official documentation:

  • Linters and Formatters

This project uses several linters and formatters to maintain code quality and consistency:

Black: A code formatter that automatically formats your code according to the PEP 8 style guide. The configuration is set in the pyproject.toml file under [].

Ruff: A linter that checks your code for style guide violations, potential bugs, and other issues. This covers so many rules inluding flake8’s one and isort’s one. The configuration is set in the pyproject.toml file under [tool.ruff].

Mypy: A static type checker that helps catch potential bugs and ensures that your code is type-safe. The configuration is set in the pyproject.toml file under [tool.mypy].

These linters and formatters are integrated into the project's tasks and the IDE settings, ensuring a consistent code style across the entire project.

2. Create Directory Structure

Create the necessary directories and files:

mkdir assets # you can write main codes in this directory
mkdir tests # you can write test codes in this directory
Enter fullscreen mode Exit fullscreen mode

3. Write Sufficient Unit Tests with pytest

Ensure that you write comprehensive unit tests using pytest for all critical components of your project. Adequate test coverage helps maintain code quality and reduces the likelihood of introducing bugs during development or refactoring.

To write a unit test, create a unit test, create a new file in the tests directory with a name that follows the pattern test_*.py.

Within the test file, import the necessary modules and define test functions that follow the pattern def test_*():.

Use pytest's built-in assert statement to compare expected and actual values.

To run tests, execute the following command:

poetry run task test
Enter fullscreen mode Exit fullscreen mode

Ensure that all tests pass before committing changes and creating a pull request.

4. Configure IDE Settings

Reflect the contents of the provided pyproject.toml file in the settings file related to your IDE.

For example, if you are using VSCode, add the following settings to .vscode/settings.json:

  "python.formatting.provider": "black",
  "python.formatting.blackArgs": ["--line-length", "102"],
  "python.linting.flake8Enabled": true,
  "python.linting.mypyEnabled": true,
  "python.sortImports.args": ["--profile=black", "--line-length=102"],
  "editor.codeActionsOnSave": {
    "source.organizeImports": true,
    "source.fixAll": true
Enter fullscreen mode Exit fullscreen mode
  • With these settings, VSCode will automatically format and lint code using Black, Flake8, Mypy, and isort.

  • If you use IntelliJ/Pycharm, Please use File Watcher function for auto-running poetry run task fmt_lint

Image description

5. Set up CI(Auto lint/test) with GitHub Actions

Create the .github/workflows directory in the project root:

mkdir -p .github/workflows

Create a test_and_lint.yml file and add the following content:

name: python lint and test

      - main
      - "YOUR_PROJECT/**.py"
      - "tests/**.py"
      - "poetry.toml"
      - "poetry.lock"
      - ".github/workflows/test_and_lint.yml"

    name: Lint and Test
    runs-on: ubuntu-latest
      fail-fast: false
        python-version: ["3.11","3.10","3.9"]

      - name: Checkout
        uses: actions/checkout@v2

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
          python-version: ${{ matrix.python-version }}

      - name: Install Poetry
        run: |
          curl -sSL | POETRY_HOME=$HOME/.poetry python3 - 
          echo "$HOME/.poetry/bin" >> $GITHUB_PATH

      # cacheがうまく効かない環境があるのでその際は以下の設定はコメントアウトする
      - uses: actions/cache@v2
        id: venv_cache
          path: .venv
          key: venv-${{ matrix.python-version }}-${{ hashFiles('**/poetry.lock') }}

      - name: Install Dependencies
        if: steps.venv_cache.outputs.cache-hit != 'true'
        run: poetry install

      - name: Python Test
        run: poetry run task test

      - name: Python Lint
        run: poetry run task lint

Enter fullscreen mode Exit fullscreen mode

6. Set up Stage/Prod Environment with Dockerfile

For the stage/prod environment, use a Dockerfile to build the execution environment. First, export the project's dependencies to a requirements.txt file using the following command:

poetry export --without-hashes -f requirements.txt -o requirements.txt
Enter fullscreen mode Exit fullscreen mode

In the Dockerfile, use pip install to install the dependencies from the requirements.txt file. Ensure that the specified Python version in the Dockerfile is the same as the one used in the local environment.

By following these rules, you can ensure a consistent development environment for all team members and maintain a clean separation between local and stage/prod environments.

Additionally, it is recommended to integrate the configuration of this process into the GitHub Actions YAML for the next 7 Continuous Deployment (CD) steps, as it eliminates the need to work locally.

7. Set up CD(Atuo build Dockerfile) with Github Actions


Top comments (0)