Deploy a Python Celery Worker

Introduction

Celery is an open-source distributed task queue that focuses on real-time processing and task scheduling. It coordinates tasks through a backing message broker, typically Redis or RabbitMQ, making it straightforward to scale as demand increases. While Celery is written in Python, its messaging protocols are not language specific, allowing for clients in any language.

In this guide, we will go over how to deploy a Celery worker on the Koyeb serverless platform using git-driven deployment. We will create a repository with Celery and its dependencies installed, deploy RabbitMQ to act as our backing message broker, and then connect our GitHub account to Koyeb to automatically deploy the Celery worker to the platform.

At the end of this guide, you should have a Celery worker deployed to Koyeb that you can automatically update by pushing changes to your GitHub account.

Requirements

To follow along with this guide, you will need the following:

• Python 3 installed on your local machine
• A Koyeb account to deploy and run your Celery worker and RabbitMQ message broker
• A GitHub account to store your application code and trigger deployments via repository changes

Steps

This guide will cover how to deploy a Celery worker to Koyeb through the following steps:

1. Create a Python Virtual Environment
2. Install Celery
3. Create a Basic Celery Application
4. Push the Celery Project to GitHub
5. Deploy RabbitMQ on Koyeb
6. Deploy the Celery Worker on Koyeb
7. Testing the Celery Worker

Create a Python Virtual Environment

To get started, we will create a Python virtual environment on our local machine to isolate our project's dependencies from the rest of the system packages. Python provides this functionality natively through the standard venv module.

Start by creating a new directory to store our project files.

mkdir ~/celery-koyeb

Next, create the virtual environment within the project directory by typing:

python3 -m venv ~/celery-koyeb/venv

This will create a directory called venv within the ~/celery-koyeb containing initialization scripts, copies of essential tools like pip, and links to the local Python interpreters.

To tell Python to use the virtual environment, activate it by sourcing the initialization script:

source ~/celery-koyeb/venv/bin/activate

Your shell prompt will change to reflect that the virtual environment is active for this session. Python will now use this directory structure in place of the system's standard Python file locations, allowing us to isolate dependencies.

Install Celery

Next, we need to install Celery within the virtual environment using pip, the Python package manager:

pip install celery

Python will download Celery and its dependencies and install them within the virtual environment filesystem.

Along with the Python modules, the installation also includes a small wrapper script called celery that lets you interact with it from the command line. Use the command now to verify that the installation was successful:

celery --version

The command should show the version as expected. The exact version returned does not matter, so long as the command returns successfully:

5.2.7 (dawn-chorus)

With Celery installed, record the project's dependencies by typing:

pip freeze > ~/celery-koyeb/requirements.txt

Create a Basic Celery Application

Next, we'll create a basic Celery application in our project directory.

Create a file called tasks.py within the ~/celery-koyeb directory with the following content inside:

import os
from celery import Celery


# Fail if the `BROKER_HOST` environment variable is not defined
BROKER_HOST = os.environ.get('BROKER_HOST', None)
if BROKER_HOST is None:
    raise Exception("BROKER_HOST environmental variable is not defined")
broker_url = 'pyamqp://guest@' + BROKER_HOST + '//'

app = Celery('tasks', broker=broker_url, backend='rpc://')

@app.task
def add(x, y):
    return x + y

Let's talk a bit about what is happening in the code above.

After importing the os module and the Celery class, the code above sets the location of the backing RabbitMQ message broker based on the value of the BROKER_HOST environment variable. If this variable is not set, we will error out to avoid accidentally deploying misconfigured instances.

Next, we create an instance of the Celery class called app. We pass tasks as the name of the main module and include the broker and backend configuration to allow Celery to connect to the RabbitMQ instance indicated by the environment variable mentioned earlier.

Finally, we create a function with the @app.task decorator to add it to Celery's task registry. Our function adds two numbers together and returns the result.

Save and close the file when you are finished.

Push the Celery Project to GitHub

Next, we will commit our project to git and push our changes to our GitHub account.

In your project directory, initialize a git repository to get started:

cd ~/celery-koyeb
git init

Add the tasks.py application and the requirements.txt file to the staging area. You can also create a minimal .gitignore file to make sure your virtual environment does not accidentally get committed. Commit the changes when you are done:

echo 'venv' >> .gitignore
git add tasks.py requirements.txt .gitignore
git commit -m 'Initial commit'

Create a new repository within your GitHub account. This can be either public or private. Add the GitHub repository address as the origin of your project and push your changes to the repository:

git remote add origin git@github.com:<YOUR_GITHUB_USERNAME>/<YOUR_GITHUB_REPOSITORY>.git
git branch -M main
git push -u origin main

If you refresh the GitHub repository page in your browser, your project files will appear.

Deploy RabbitMQ on Koyeb

Before we deploy our Celery worker, we need to deploy a RabbitMQ instance for Celery to use as a message broker. We will use the official RabbitMQ image on DockerHub for this purpose.

Go to the Koyeb control panel and click the Create App button. Select Docker as the deployment method to continue.

On the next screen, in the Image field, type docker.io/rabbitmq and click Next.

Next, choose Web Service as your service type and select your desired region and instance size. Click Advanced to expand the networking configuration options.

In the Environment variables section, change the value of the PORT variable to RabbitMQ's default port: 5672.

In the Exposing your service section, change the port number to 5672 and deselect the Public option. This will allow other services within this application (like our Celery application) to access the message broker using its service name without exposing the service to the internet.

When you have completed the above configuration, click the Deploy button.

Your new application will be created and the RabbitMQ service will begin to deploy within it. In a few minutes, the RabbitMQ container will finish deploying to Koyeb and the service will be marked as healthy.

Before moving on, copy the Private domain of your RabbitMQ deployment. We will need this later as we configure the Celery worker service.

Deploy the Celery Worker on Koyeb

Now that the RabbitMQ service is deployed, we can deploy the Celery application from our GitHub repository.

In the Koyeb control panel, click the app that contains your RabbitMQ service. Click the Create Service button to deploy a new service to the app alongside the RabbitMQ container. Select GitHub as the deployment method to continue.

Note: If this is your first time using Koyeb with GitHub, click the Install GitHub App button to connect your accounts. On the authorization page, select which repositories you want Koyeb to access and click the Install button to continue. You will be redirected back to Koyeb to continue your deployment.

Next, select the GitHub repository for your Celery application to continue.

On the following page, click Build and deployment settings to expand the options for your deployment. Click the Override toggle associated with the Run command field so that we can specify the command we want to run in this deployment. In the Run command field, type:

celery --app=tasks.app worker

This will start a Celery worker using the app instance we configured within the tasks.py file.

Next, select Worker as the service type and select your desired region and instance size. Click Advanced to expand additional configuration options.

In the Environment variables section, create a new variable called BROKER_HOST to tell the Celery worker where to find the RabbitMQ instance. Set the value to the Private domain you copied from the RabbitMQ service. It will follow this format:

<YOUR_SERVICE_NAME>.<YOUR_DEPLOYMENT_NAME>.koyeb

When you have completed the above steps, click the Deploy button.

The Celery application will be pulled from your GitHub repository and deployed. In a few minutes, the Celery worker will begin running within the service and be available for work.

If you want to learn about how Koyeb automatically builds your Python applications from git, make sure to read the how we build from git documentation.

Testing the Celery Worker

Unlike a web application, Celery does not have a web interface to easily show you that the deployment succeeded. We can, however, use Koyeb's built-in Console to test that the worker is running, able to connect with RabbitMQ, and can process tasks.

In the Koyeb control panel, click the Celery service and then select the Console tab.

In the console window, start a bash session by typing:

/bin/bash

Next, source the environment configuration file so that Python can find the modules associated with our application:

source .profile.d/python.sh

Start a Python session so that we can interact with the Celery application:

python

First, import our tasks module:

import tasks

Next, we can run the add function using Celery's delay method:

results = tasks.add.delay(93, 80)

We can then use the results object to check on the task's progress. Call the ready method to see if it has been processed yet:

results.ready()
True

You can check the answer returned by the function using the get method. It's usually a good idea to set a timeout so that it won't hang waiting if the result is not available yet:

results.get(timeout=1)
173

While this test uses trivial calculations, the results validate that the system is working as intended and able to communicate with RabbitMQ using Koyeb's service mesh.

When you are finished, exit the Python session by typing:

exit()

You can then close the bash session by typing:

exit

Conclusion

In this guide, we covered how to deploy a Celery worker on Koyeb using git-driven deployment. Because Celery requires a message broker for coordination, we also discussed how to deploy a RabbitMQ container on Koyeb using Docker-based deployment.

By deploying both of these services within a single application, we were able to configure Celery to connect with RabbitMQ over Koyeb's service mesh with minimal configuration. Our Celery deployment is backed by a GitHub repository, so it will automatically redeploy whenever new changes are pushed to the branch it is tracking. This makes it easy to change your running Celery worker configuration by modifying your project's code.

Questions or suggestions to improve this guide? Join us on the community platform to chat!