Asynchronous tasks in Django with Django Q

Learn how to use Django Q, the task queue, with the Redis broker for offloading long running tasks in your Django applications.

Requirements

To follow along you'll need:

• an Heroku account if you want to use their Redis add-on
• the Heroku CLI installed on your system
• a newer version of Python, ideally 3.6 or 3.7
• Git

Deployment on Heroku is optional, and you can use your own Redis instance if you've already got one locally.

Setting up the project

And now let's get to work! To start off we're going to create a new Python virtual environment alongside with a Django installation:

mkdir django-q-django && cd $_
python3 -m venv venv
source venv/bin/activate
pip install django

Next up we're going to create a new Django project from a template:

django-admin startproject \
    --template https://github.com/valentinogagliardi/ponee/archive/master.zip \
    --name=Procfile \
    --extension=py,example django_q_django .

If you're wondering what I'm doing here, this is a template of mine. I've got a link in the resources with a tutorial for creating your own Django project template.

Now let's install the dependencies with pip:

pip install -r ./requirements/dev.txt

We need also to provide some environment variables for our project:

mv .env.example .env

and finally we're going to run Django migrations:

python manage.py makemigrations
python manage.py migrate

at this point you should be able to run the development server:

python manage.py runserver

Now before moving to Django Q let's see what problem is it meant to solve.

Asynchronous tasks in Django with Django Q: the problem with synchronous code

The main issue for Python and Django is that they're synchronous. It's not a bad thing per se, and there are a lot of ways to circumvent it.

Python, on which Django builds on, is single threaded by nature. Single threaded means that the language interpreter can only run your code in sequence.

The practical implication is that any view in a Django application can get stuck if one or more operations take too much to complete.

To demonstrate the concept let's create a new Django application inside our project:

django-admin startapp demo_app

In this app we're going to define a view which returns a simple JSON response:

# demo_app/views.py

from django.http import JsonResponse

def index(request):
    json_payload = {
        "message": "Hello world!"
    }
    return JsonResponse(json_payload)

And let's also create the corresponding url:

# demo_app/urls.py

from django.urls import path
from .views import index

urlpatterns = [
    path("demo-app/", index)
]

Don't forget to wire up the url for the new app:

# django_q_django/urls.py

from django.contrib import admin
from django.urls import path, include
from .settings.base import ADMIN_URL

urlpatterns = [
    path(f'{ADMIN_URL}/', admin.site.urls),
    # the new url
    path("", include("demo_app.urls"))
]

And finally activate the app:

# django_q_django/settings/base.py

INSTALLED_APPS = [
    # omitted for brevity
    'demo_app.apps.DemoAppConfig'
]

Now to simulate a blocking event in the view we're going to use sleep from the time module, part of the Python standard library:

from django.http import JsonResponse
from time import sleep

def index(request):
    json_payload = {
        "message": "Hello world!"
    }
    sleep(10)
    return JsonResponse(json_payload)

Run the development server, head over http://127.0.0.1:8000/demo-app/ and you can see the view hanging for 10 seconds before returning to the user.

Now, this is a delay created on purpose, but in a real application the block could happen for a number of reasons:

• I/O operations taking too long
• network delay
• interactions with the file system

Even if it's a contrived example you can see why it's crucial to offload long running tasks in a web application.

Django Q was born with this goal in mind. In the next sections we'll finally put our hands on it.

Wait, how about asynchronous Django?

We're left with a Django project, a Django application, and a view that remains stuck for 10 seconds!

It's not that Django and Python don't scale. There are a number of ways to get around single threading.

For Python there is asyncio. Django instead moved to async only recently, the implementation is in its infancy and still there's no support for async views.

Things are going to change in the future, I suggest staying tuned on Andrew Godwin because it's the lead person for the async story in Django.

But the need for third party queues won't go away anytime soon even when Django will go 100% async. Still a nice skill to have.

Preparing the Heroku app and the Redis instance

In this section we'll prepare the Heroku project. I'm using Heroku here because you may want to deploy to production later, and also because they offer the Redis add-on for free.

If you're new to Redis, it's an in-memory database, can be used as a cache and as a message broker.

A message broker is more or less like a post office box: it takes messages, holds them in a queue, and folks from around the city can retrieve these messages later.

If you're interested in how Django Q uses brokers check out this page.

Still in the project folder initialize a Git repo:

git init

Then create a new Heroku app. I'm going to add two add-ons:

• heroku-postgresql which is more robust than the default sqlite for production
• heroku-redis which will give us the Redis instance

If you haven't got the Heroku CLI and an Heroku account go create one, install the CLI and come back later.

Otherwise follow along with me and create the app:

heroku create --addons=heroku-postgresql,heroku-redis

Once done give Heroku a couple of minutes and then run:

heroku config:get REDIS_URL

This command will reveal REDIS_URL, an environment variable with the credentials for the Redis instance.

Take note of it and head over the next section!

Asynchronous tasks in Django with Django Q: installing and running Django Q

Let's install Django Q and the Redis client library (the client is needed by the Redis broker for Django Q):

pip install django-q redis

Once done activate Django Q in the list of installed apps:

INSTALLED_APPS = [
    # omit
    # add Django Q
    'django_q'
]

Now reveal the Redis Heroku credentials:

heroku config:get REDIS_URL

You should see a string like this:

redis://h:p948710311f252a334c3b21cabe0bd63f943f68f0824cd41932781e7793c785bf@ec2-52-18-11-1.eu-west-1.compute.amazonaws.com:9059

Before the @ you'll find the password:

p948710311f252a334c3b21cabe0bd63f943f68f0824cd41932781e7793c785bf

After the @ there's the host:

ec2-52-18-11-1.eu-west-1.compute.amazonaws.com

And 9059 is the port. Note that the credentials will be different for you, don't use mine!

(Needless to say, by the time you read this article these credentials will be gone.)

Now configure Django Q in django_q_django/settings/base.py. Fill host, port, and password with your credentials:

Q_CLUSTER = {
    'name': 'django_q_django',
    'workers': 8,
    'recycle': 500,
    'timeout': 60,
    'compress': True,
    'save_limit': 250,
    'queue_limit': 500,
    'cpu_affinity': 1,
    'label': 'Django Q',
    'redis': {
        'host': 'ec2-52-18-11-1.eu-west-1.compute.amazonaws.com',
        'port': 9059,
        'password': 'p948710311f252a334c3b21cabe0bd63f943f68f0824cd41932781e7793c785bf',
        'db': 0, }
}

You might wonder why I'm not using REDIS_URL as it is. The reason is that Django Q wants credentials in a dictionary.

I didn't have time to check if is the Python Redis client imposing this limitation, maybe I'll write a patch for both in the future. It was a limitation of Django Q, hope I'll have time to open a PR I opened a pull request which got merged, and now you can use a Redis url:

Q_CLUSTER = {
    'name': 'django_q_django',
    # omitted for brevity  
    'label': 'Django Q',
    'redis': 'redis://h:asdfqwer1234asdf@ec2-111-1-1-1.compute-1.amazonaws.com:111'
}

(When running the project in production you may want to switch to using environment variables. See the base configuration for learning how to use env).

Once you're done run the migrations (Django Q needs to create its tables in the database):

python manage.py migrate

At this point you're ready to run the Django Q cluster with:

python manage.py qcluster

If everything goes well you should see this:

Well done! In the next section we'll create our first asynchronous task.

What is the Django Q cluster? Check this out.

Asynchronous tasks in Django with Django Q: async_task

Worth doing a quick recap of what we covered so far:

• we created a Django project
• we created a Django application
• we installed Django Q and the Redis client
• we created an Heroku project and a Redis instance
• finally we configured Django Q

To test that Django Q could connect to Redis I launched:

python manage.py qcluster

With the project in place let's finally see an example of Django Q in action. Remember your view?

# demo_app/views.py

from django.http import JsonResponse
from time import sleep

def index(request):
    json_payload = {
        "message": "Hello world!"
    }
    sleep(10)
    return JsonResponse(json_payload)

Remove the time import and create a new file in demo_app/services.py (the name of this file is totally up to you).

In this new module we're going to define a function, sleep_and_print:

# demo_app/services.py

from time import sleep

def sleep_and_print(secs):
    sleep(secs)
    print("Task ran!")

In the view instead we'll borrow async_task from Django Q:

from django.http import JsonResponse
from django_q.tasks import async_task


def index(request):
    json_payload = {
        "message": "hello world!"
    }
    """
    TODO
    """
    return JsonResponse(json_payload)

async_task is the principal function you'll use with Django Q. It takes at least one argument, the function's module that you want to enqueue:

# example

async_task("demo_app.services.sleep_and_print")

The second group of arguments instead is any argument that the function is supposed to take. sleep_and_print in our example takes one argument, the seconds to wait before printing. That means for async_task:

# example

async_task("demo_app.services.sleep_and_print", 10)

That's enough to enqueue a task. Let's now mix our view with async_task.

Asynchronous tasks in Django with Django Q: enqueue your first task

Back to our view, with async_task imported, call it right after the return statement:

from django.http import JsonResponse
from django_q.tasks import async_task


def index(request):
    json_payload = {"message": "hello world!"}
    # enqueue the task
    async_task("demo_app.services.sleep_and_print", 10)
    #
    return JsonResponse(json_payload)

Now run the cluster:

python manage.py qcluster

Run the Django server:

python manage.py runserver

And finally make a call to your view, either from http://127.0.0.1:8000/demo-app/ or from the terminal:

curl http://127.0.0.1:8000/demo-app/

Now you should notice a couple of things. The Django dev server should log:

13:55:42 [Q] INFO Enqueued 1

The Django Q cluster should log something along these lines:

13:55:42 [Q] INFO Process-1:1 processing [juliet-mountain-august-alaska]

And after that you should see:

Task ran!

Here's my terminal:

What happened here is that:

1. the Django view responded immediately to the request
2. Django Q saved the task (just a reference) in Redis
3. Django Q ran the task

With this "architecture" the view does not remain stuck anymore. Brilliant.

Think about the use cases for this pattern. You can:

• safely interact with the I/O
• crunch data in the background
• safely move out API calls from your views

and much more.

Asynchronous tasks in Django with Django Q: what's next?

In addition to async_task Django Q has the ability to schedule a task. A practical use case is do X every X days, much like a cron job. Check the documentation to learn more.

Django Q supports other brokers in addition to Redis. Again, the docs are your friend.

If you don't need other brokers than Redis, django-rq might be a lightweight alternative to Django Q.

Asynchronous tasks in Django with Django Q: why not Celery?

Fun fact: Celery was created by a friend of mine. We were in high school together. Despite that I don't have much experience with Celery itself, but I always heard a lot of people complaining about it.

Check this out for a better perspective.

Thanks for reading and stay tuned!

Resources

• Django Q
• Redis
• What is message broker?
• Background Jobs and Queueing
• How to create a Django project from a template

Originally published on my blog