DEV Community

Cover image for Experimentations on Bazel: Python (1), FastAPI
David Bernard
David Bernard

Posted on • Edited on

Experimentations on Bazel: Python (1), FastAPI

#bazel #python #fastapi

Goals

Setup a python project with:

  • split the code into packages (eg web, services, models,...)
  • use FastAPI as web framework to handle http request
  • use pytest for test the code
  • format the code with black
  • check/audit the code with mypy and other linters (managed by [pylama]
  • enforce the python version use to build, check,...
  • integration with an IDE/Editor (VSCode in my case)

Steps

Setup python rules

Follow instructions from bazelbuild/rules_python

Add into WORKSPACE.bazel

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")


#------------------------------------------------------------------------------
# Python
#------------------------------------------------------------------------------

# enable python rules
http_archive(
    name = "rules_python",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.2.0/rules_python-0.2.0.tar.gz",
    sha256 = "778197e26c5fbeb07ac2a2c5ae405b30f6cb7ad1f5510ea6fdac03bded96cc6f",
)
Enter fullscreen mode Exit fullscreen mode

ℹ️ In every starlark source file (the language used for *.bazel, *.bzl), to be able to use a function, you should start by using load(<from>, <function_a>, <function_b>,...), it's like import or use in other programming language.

To prepare usage of external dependencies (like fastapi, pytest,...), create the file third_party/requirements.txt.

mkdir third_party
touch third_party/BUILD.bazel
cat >third_party/requirements.txt <<EOF
# list externals dependencies available for every python packages
EOF
Enter fullscreen mode Exit fullscreen mode

Update WORKSPACE.bazel to create the repo with externals dependencies for every python packages.

# Create a central repo that knows about the dependencies needed for
# requirements.txt.
load("@rules_python//python:pip.bzl", "pip_install")

pip_install(
    name = "my_python_deps",
    requirements = "//third_party:requirements.txt",
)
Enter fullscreen mode Exit fullscreen mode

But this setup use the python installed on your local environment, and we want to enforce the python version that will be used. I found 2 alternatives:

Both have pros & cons, because we'll use pyenv for integration with IDE/Editor, go for the second (it's main downside is to install 2 versions of python, because python rules should know interpreter for python 2.x and 3.x). So update WORKSPACE.bazel with

# use pyenv to enforce python version
http_archive(
    name = "dpu_rules_pyenv",
    sha256 = "d057168a757efa74e6345edd4776a1c0f38134c2d48eea4f3ef4783e1ea2cb0f",
    strip_prefix = "rules_pyenv-0.1.4",
    urls = ["https://github.com/digital-plumbers-union/rules_pyenv/archive/v0.1.4.tar.gz"],
)

load("@dpu_rules_pyenv//pyenv:defs.bzl", "pyenv_install")

pyenv_install(
    hermetic = False,
    py2 = "2.7.18",
    py3 = "3.9.2",
)
Enter fullscreen mode Exit fullscreen mode

Add FastAPI code

Create the file exp_python/webapp/main.py with

from fastapi import FastAPI

app = FastAPI()


@app.get("/status")
def read_root():
    return {"status": "UP", "version": "0.1.0"}
Enter fullscreen mode Exit fullscreen mode

Create the file exp_python/webapp/BUILD.bazel with

load("@rules_python//python:defs.bzl", "py_library")
load("@my_python_deps//:requirements.bzl", "requirement")

py_library(
    name = "webapp",
    srcs = ["main.py"],
    srcs_version = "PY3",
    deps = [requirement("fastapi")],
)
Enter fullscreen mode Exit fullscreen mode

If you try to build now bazel build //exp_python/webapp, you will have an error with:

ERROR: /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/BUILD.bazel:4:11: no such package '@my_python_deps//pypi__fastapi': BUILD file not found in directory 'pypi__fastapi' of external repository @my_python_deps. Add a BUILD file to a directory to mark it as a package. and referenced by '//exp_python/webapp:webapp'
Enter fullscreen mode Exit fullscreen mode

Because requirement("fastapi") is not defined, to fix this, update third_party/requirements.txtchange

# list externals dependencies available for every python packages
fastapi==0.63.0
Enter fullscreen mode Exit fullscreen mode

Test webapp

Add exp_python/webapp/test.py, with a code similar to the FastAPI guide, but with a assert True == False at the end, so the test will failed (because no error could also mean that test are not launch, especially when you're setup the test flow).

from fastapi.testclient import TestClient

from .main import app

client = TestClient(app)


def test_read_main():
    response = client.get("/status")
    assert response.status_code == 200
    assert response.json() == {"status": "UP", "version": "0.1.0"}
    assert True == False
Enter fullscreen mode Exit fullscreen mode

And modify BUILD.bazel to use py_test

load("@rules_python//python:defs.bzl", "py_library", "py_test")
load("@my_python_deps//:requirements.bzl", "requirement")

...

py_test(
    name = "test",
    srcs = [
        "test.py",
    ],
    # main = "test.py",
    python_version = "PY3",
    srcs_version = "PY3",
)
Enter fullscreen mode Exit fullscreen mode

Calling bazel test //exp_python/webapp:test failed but said to look into a file under bazel-out to see the log. It's not convenient, so configure bazel to output error to console, by adding into .bazelrc (see previous article) the default cli option for test to display errors on stdout

test --test_output=errors
Enter fullscreen mode Exit fullscreen mode

Now we can see the error on stdout: No module named 'fastapi'. In fact to be able to run test, we should add as dependencies for the test:

  • :webapp to be able to access the SUT (system under test)
  • fastapi because test import it (in fact it is transitively available through :webapp)
  • requests, maybe it's a bug; it's a dependencies of fastapi > starlette for testing but it doesn't seems to be transitively available.
  • pytest because without, the test was not launch (always green) 
py_test(
    name = "test",
    srcs = [
        "test.py",
    ],
    # main = "test.py",
    args = [
        "--capture=no",
    ],
    python_version = "PY3",
    srcs_version = "PY3",
    deps = [
        ":webapp",
        requirement("requests"),
        requirement("fastapi"),
        requirement("pytest"),
    ],
)
Enter fullscreen mode Exit fullscreen mode

third_party/requirements.txt

# list externals dependencies available for every python packages
fastapi==0.63.0

#test
requests==2.25.1
pytest==6.1.2
Enter fullscreen mode Exit fullscreen mode

In fact, when running the test the command launch is something like python test.py with test.py id the value from the main argument of py_test (by default the value of name + .py) and it should be a member of srcs. So no way to call the module pytest. The workaround is to call pytest from test.py.

from fastapi.testclient import TestClient

from exp_python.webapp.main import app

client = TestClient(app)


def test_read_main():
    response = client.get("/status")
    assert response.status_code == 200
    assert response.json() == {"status": "UP", "version": "0.1.0"}
    assert True == False

# if using 'bazel test ...'
if __name__ == "__main__":
    import sys
    import pytest
    sys.exit(pytest.main([__file__] + sys.argv[1:]))
Enter fullscreen mode Exit fullscreen mode

Notice that app is now imported from exp_python.webapp.main and no longer relative. I didn't find how to handle it, without providing the full package name from root workspace.

Now when launch test bazel test //exp_python/webapp:test, it failed as expected (I let you fix the test)

INFO: From Testing //exp_python/webapp:test:
==================== Test output for //exp_python/webapp:test:
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/53/execroot/__main__/bazel-out/k8-fastbuild/bin/exp_python/webapp/test.runfiles/__main__
collected 1 item

exp_python/webapp/test.py F

=================================== FAILURES ===================================
________________________________ test_read_main ________________________________

    def test_read_main():
        response = client.get("/status")
        assert response.status_code == 200
        assert response.json() == {"status": "UP", "version": "0.1.0"}
>       assert True == False
E       assert True == False

exp_python/webapp/test.py:12: AssertionError
=========================== short test summary info ============================
FAILED exp_python/webapp/test.py::test_read_main - assert True == False
============================== 1 failed in 0.10s ===============================
================================================================================
Target //exp_python/webapp:test up-to-date:
  bazel-bin/exp_python/webapp/test
INFO: Elapsed time: 0.923s, Critical Path: 0.85s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//exp_python/webapp:test                                                 FAILED in 0.8s
  /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/execroot/__main__/bazel-out/k8-fastbuild/testlogs/exp_python/webapp/test/test.log

INFO: Build completed, 1 test FAILED, 2 total actions
Enter fullscreen mode Exit fullscreen mode

Run webapp

The goal is also to be able to run the webapp. To launch a FastAPI webapp, the recommended way is to use uvicorn

# list externals dependencies available for every python packages
fastapi==0.63.0
uvicorn==0.13.4

#test
requests==2.25.1
pytest==6.1.2
Enter fullscreen mode Exit fullscreen mode

But we can not launch uvicorn from command line, because we do not install it "globaly" on the system. On the other side, our target is to create an executable, that we can later launch on local or into a container.

Building executable is the goal of rule with suffix _binary in the Bazel ecosystem like py_binary.

load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
load("@my_python_deps//:requirements.bzl", "requirement")

...

# to add additionals parameters place them after "--" in bazel call, like:
# `bazel run //exp_python/webapp:run -- --reload`
py_binary(
    name = "run",
    srcs = ["run.py"],
    python_version = "PY3",
    srcs_version = "PY3",
    visibility = ["//visibility:public"],
    deps = [
        ":webapp",
        requirement("uvicorn"),
    ],
)
Enter fullscreen mode Exit fullscreen mode

As you can see, we also introduce run.py, because like for py_test previously there is main attribute and only file from srcs can be used.

So create exp_python/webapp/run.py

import uvicorn
import sys

if __name__ == '__main__':
    # freeze_support()
    sys.argv.insert(1, "exp_python.webapp.main:app")
    sys.exit(uvicorn.main())
Enter fullscreen mode Exit fullscreen mode

And try it

bazel run //exp_python/webapp:run
# open into browser or via curl http://127.0.0.1:8000/status
Enter fullscreen mode Exit fullscreen mode

If you launch with bazel run //exp_python/webapp:run -- --reload and do change into main.py they should be detected and apply.

Editor

At this point, we have a bazel project that is working, but bazel is not really well supported by IDE / Editor (except for edition of bazel configuration file and sometimes launch of command).
What we can do to improve a little, is to create a python virtual env with same python version and python external dependencies (not scoped by packages, target, usage).

Currently I don't know, how to do it better (suggestions are welcomes), so we'll create a file at the root of the workspace setup_localdev.sh:

#!/bin/bash

PYENV_VERSION="3.9.2"

eval "$(pyenv init -)"
pyenv install ${PYENV_VERSION} --skip-existing
pyenv local ${PYENV_VERSION}
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r third_party/requirements.txt
python --version
Enter fullscreen mode Exit fullscreen mode

This file could be generated by bazel, but I failed to :

  • find a way to share the python version with WORKSPACE.bazel
  • generate the shell script with eval "$(pyenv init -)" (Maybe a suject for an other article)

Run this script, configure your editor to use the virtual environement .venv and continue to edit code. Re-run the script everytime you update requirements.txt (remove of dependencies, do not clean the virtual environment).

To be continued

It's not the end, we have more stuff to setup (linters,...), but we're in a state enough to work.

The sandbox_bazel is hosted on github (not with the same history, due to errors), use tag to have the expected view at end of article: article/4_python_1. I'll be happy to have your comments on this article, or to discuss on github repo.

Top comments (1)

Collapse
 
joeleche profile image
Joel Eche •

Hi, thank you for you guide
I have a question, I was following the notes and I got this error

my_python_deps/pypi__dataclasses/dataclasses.py", line 550, in _is_classvar
    return type(a_type) is typing._ClassVar
AttributeError: module 'typing' has no attribute '_ClassVar'
Enter fullscreen mode Exit fullscreen mode

I read in other sites that it can be fixed uninstalling dataclasses, however I don't understand too much how to run commands like "pip uninstall ..." on a builded project. Do you have any idea about how to fix it?

Read next

furkangozukara profile image

How to Set Up Python, CUDA, cuDNN, C++ Build Tools, FFMPEG & Git for AI Applications

Furkan Gözükara -

mameen profile image

Understanding the Factory and Factory Method Design Patterns

Ahmad Ameen -

tala profile image

Supercharge Your Conversations: How to Build a Telegram Bot with OpenAI's GPT-4 (No-Code!)

Tala -

garciadiazjaime profile image

Bridging Machine Learning with TensorFlow: From Python to JavaScript

Jaime -