Django, Celery, and Redis on Railway

Introduction

Using Django Rest Framework, Celery, and Redis I created a personalized quiz to help people find roles in tech they would like. The app asks you questions and creates a custom image with your results that needs time to process in the background, so I leverage Celery and Redis to get this work done!

Now the app took me a day to build, but a week to deploy to production. Why? The age old story of hard-to-find, or nonexistent documentation. To break the broken cycle in tech I’ve created this tutorial to cover the following steps:

1. How to deploy and set up Redis on Railway
2. How to connect your Django server to your Redis server on Railway
3. How to run your Celery workers during the build

This blog will not cover:

1. The basics of Python
2. Getting started with Django Rest Framework
3. Getting started with Celery
4. Getting started with Redis

Prerequisites

1. Skills needed: Basic knowledge of Django, Redis, Celery
2. Tools required: A Django Project that’s set up to use Redis & Celery

Or you can just read along for fun if you’re not going to build with the tutorial!

Section 2: Project Overview

1. Below is a simplified overview of the project structure so you can have a better understanding of how the project is structured
   1. For this blog, I’ve bolded the files that we’ll focus on to make sure

tbc/
├── manage.py
├── ➕ Procfile
├── tbc/
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── ➕ celery.py
│   └── wsgi.py
└── quiz/
    ├── migrations/
    │   └── __init__.py
    ├── __init__.py
    ├── admin.py
    ├── apps.py
    ├── models.py
    ├── serializers.py
    ├── ➕ tasks.py
    ├── tests.py
    ├── urls.py
    └── views.py

celery.py

import os

from celery import Celery

# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tbc.settings')

app = Celery('tasks',
             backend=os.getenv('REDIS_URL'),
             broker=os.getenv('REDIS_URL')
             )

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django apps.
app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print(f'Request: {self.request!r}')

Setting Up Railway

Create a Redis Server in Railways

Now that the files are all ready for different environments, let’s head over to the project in Railway to add a new Redis instance.

1. To get started, head over to your Railway dashboard and select the project you would like to add your Redis setup to. Once in the project, click '+ New' to get the service dropdown.
   

2. From the dropdown, you want to select Redis by either typing in Redis to have it pop up or selecting the database to find the Redis option.
   

Adding a Railways Reference Variable

In our code, we use the REDIS_URL environment variable in our [celery.py](http://celeary.py) file so our project can use different Redis URLs for each environment we’re working in.

For the production environment, we want our REDIS_URL to connect to the Redis server we created in the previous step. To do this we leverage the Railway Reference variable for Redis_URL.

ℹ️ The way I think of Railway Reference variables is a quick way to add variables for Railway-based services without having to type or copy and paste the value because the Railway defines the value for you! Want to learn more about this topic? Check out their documentation on variables here.

Running and Monitoring Celery Tasks In Railway

1. To run the Celery worker, you’ll need to update your Procfile to look like this:
   

   web: celery -A tbc worker --loglevel=info & python manage.py migrate && gunicorn tbc.wsgi  --bind 0.0.0.0:$PORT
   

   What is this doing?
   This allows Railway to run the celery worker when the system deploys the site. Without this task running your Django app won’t be able to connect with the celery worker.

   ℹ️ You may be thinking “hey, I have to start the Redis server when I’m running this on my local. Don’t I have to do the same for production?” And the answer is no, because Railway does it for you.

Why are we updating the Procfile file?

When working in our local files we start our Django and Redis server before we run our celery worker and make sure we run all of our migrations.

What is a Procfile file?
The Procfile is how we tell the server how to run these commands and more after Railway kicks off the server.

What does the command we add mean?
By adding celery -A tbc worker --loglevel=info & before the other commands you’re telling the Railway to run the Celery so your task can run. One thing to call out here is the order of tasks is important. Add the celery details first to avoid possible build errors you could run into if celery isn’t running during the app building.

ℹ️ If you’re familiar with Procfile files, you might be used to seeing the file set up like it is below with the celery details under the worker. The reason we’re placing the worker details on the web line is because Railway only reads commands under the web.

```python
web: python manage.py migrate && gunicorn tbc.wsgi
worker: celery -A tbc worker --loglevel=info
```

Ready for Testing

At this stage everything is connected and ready to run to be tested in production! If you've added any logging to any of your task, you'll see this information show up in your Django server in railway.

If you hit all of your workflows and everything is working as expected, congratulations because you did it!

Conclusion

In this blog we’ve set up our prod Railway environment to work with Django, Redis, and Celery!

1. We’ve been able to cover
   1. How to set up our project to support the production environment
   2. Cover the steps needed to create a Redis server in Railway
   3. How to connect your Django server in production to your Redis server

Further Resources

1. Links to documentation and more advanced guides
   1. Redis Documentation
   2. Railway Documentation
   3. Django Celery Redis Guide I found helpful

If you made it this far, I want to just say thanks for following along! Would love to hear your feedback and thoughts on this blog post.