Django Background Tasks

Background tasks are processes that run in the background so your server is ready to take further requests.

Some common use cases for background tasks:

• Web Scraping
• Heavy data processing
• Image processing
• Multiple 3rd party API calls
• Building Slack-bots (Slack requires a 3 second response)

Basically, if the task takes a long time to complete (e.g. >30 seconds) you probably want to offload it to a background task.

If you have your Django project set up it's dead easy to get a background tasks up and running.

Step 1: Getting Started

We want to install Django Q via:

pip install django-q

Django Q allows us to set up some queues to run in the background. Be sure to run the migrations as well to generate the job models (which you'll see in your admin panel)

python manage.py migrate

It requires a broker though...

Step 2: Setting up Redis

We'll use redis. This is basically the communication channel between your django instance (normal django) and your django worker (this is your background task/worker).

You can check if your redis server is running by running:

redis-cli ping

If not you'll need to follow the redis instructions to set up on your system.

Step 3: Setting up the Cluster

The standard django-q configuration will work for us locally but with a couple of changes so we can run this in production as well. Add the following to your base settings.py file:

REDIS_HOST = os.environ.get("REDIS_HOST", "localhost")
REDIS_PORT = os.environ.get("REDIS_PORT", 6379)
REDIS_DB = os.environ.get("REDIS_DB", 0)
REDIS_PASSWORD = os.environ.get("REDIS_PASSWORD", None)

Q_CLUSTER = {
    "name": "hello-django",
    "workers": 8,
    "recycle": 500,
    "timeout": 60,
    "compress": True,
    "cpu_affinity": 1,
    "save_limit": 250,
    "queue_limit": 500,
    "label": "Django Q",
    "redis": {
        "host": REDIS_HOST,
        "port": REDIS_PORT,
        "db": REDIS_DB,
        "password": REDIS_PASSWORD,
    },
}

You can play around with the number of workers - the default is 8 and these are the number of instances that can run in parallel with each other.

We can now actually start running the cluster. Have 2 terminal windows and in 1 have your standard python manage.py runserver running and in the other run:

python manage.py qcluster

You now have your cluster running.

Step 4: Creating our Task

Create a standard API to make a simple POST request and let's add the task:

from django_q.tasks import async_task
from rest_framework.views import APIView
import time

...

class SimpleTask(APIView):

    def task(self, s = 5):
        print("⏳ Running task")
        # We could call some long running code here
        time.sleep(s)
        print("⌛ Task finished")

    def post(self, request):
        # We can run the task in the background
        async_task(self.task, s=5)
        return Response({
            "message": "Task started"
        })

We use async_task to wrap the task. The code will essentially see this and send it to the redis server (which will send it to the cluster) and continue to the response.

You can place anything in the task and the response will still be returned in the same amount of time.

Step 5: Deploying to Production

Having this run in a safe local environment is one thing. Deploying it to a production environment is another.

We'll be using Railway (other than them I'd recommend Fly.io or Render - the principles are the same).

We want to add a railway.json file in our Django App with the following:

{
  "$schema": "https://railway.app/railway.schema.json",
  "build": {
    "builder": "NIXPACKS"
  },
  "deploy": {
    "restartPolicyType": "ON_FAILURE",
    "restartPolicyMaxRetries": 10
  }
}

In Railway you want to create 1 Redis database, 2 Django Apps. In the first Django App you want the Deploy (Start Command) to be:

python manage.py migrate && gunicorn NAMEOFYOURAPP.wsgi

And in the second:

python manage.py migrate && python manage.py qcluster

You'll want to update the environment variables for both applications to match the REDIS environment variables (see Step 2), for this I recommend using Railway's shared variables. The end result looks something like:

Think of this mirroring your terminals (the standard Django instance and the new Django qcluster instance)

To verify this is all working, make a request to your SimpleTask (you can find the URL if you click on the api app) and take a look at the Observability tab and you should see your background task running 🎉