After many iterations, this is my current process to deploy a django application.
The cool thing is that I'm now deploying with only an .env file and nothing else.
Note that this is just a single instance of the stack, without Kubernetes or any kind of load balancer.
Introduction
My stack consists on:
- A Postgres database
- A Redis database
- A django instance
- Celery beat and workers
I won't go into details of how to set up anything like this here, if you want to learn more about celery in their docs.
Project Structure
To understand the docker-compose.yml file below, it's important to see how I structure my django project:
MY-DJANGO-PROJECT/
├── core # settings.py lives here
├── app1/
│ ├── migrations
│ ├── models.py
│ └── ...
├── app2/
│ ├── migrations
│ ├── models.py
│ └── ...
├── data # A data directory where I store stuff like logs
├── nginx/
│ ├── certs/
│ │ ├── fullchain.pem
│ │ └── privkey.pem
│ ├── conf/
│ │ ├── default.conf
│ │ ├── prod.conf
│ │ └── staging.conf
│ └── Dockerfile
├── Dockerfile
├── entrypoint-django.sh
├── entrypoint-beat.sh
├── entrypoint-worker.sh
├── Pipfile
└── ...
Stack Diagram
To visualize the flow better here's a diagram that describes how everything is interconnected:
Docker images
The stack, though it might seem complicated, is only composed of 3 images, 2 of which are custom:
Django image
This is a custom image built from python.
This image will be used for django, celery workers and celery beat containers.
Here's the Dockerfile for it:
# Python as the base image
# I use bullseye because I'm more comfortable with it
# but you can use Alpine for a more lightweight container
FROM python:3.11-bullseye
# Exposes port 8000
# Make sure to change this to your used port
EXPOSE 8000
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Working directory
WORKDIR /app/backend
# Install pipenv
# This is not necessary if you use pip in your code
RUN pip install -U pipenv
# Install pipenv requirements
# Turns the Pipfile to a requirements.txt
# so it can be installed globally with pip
COPY Pipfile Pipfile.lock /app/backend/
RUN pipenv requirements > requirements.txt
RUN pip install -r requirements.txt
RUN rm -rf ./Pipfile ./Pipfile.lock
# Copy all the code over
COPY . .
# Create the media directory
RUN mkdir -p /app/backend/media
# Create a volume for the media directory
VOLUME /app/backend/media
# Create a volume for the static directory
VOLUME /app/backend/django_static
# Make the entrypoint scripts executable
# There's one entrypoint for each service that uses this image
RUN chmod +x /app/backend/entrypoint-django.sh
RUN chmod +x /app/backend/entrypoint-worker.sh
RUN chmod +x /app/backend/entrypoint-beat.sh
# Set the default entrypoint in case this Dockerfile is run
# by itself
ENTRYPOINT ["/app/backend/entrypoint-django.sh"]
These are the entry point files for each service:
django entry point
#!/bin/bash
# Migrate any new migrations to the database on deployment
echo "Migrating..."
python manage.py migrate --no-input
# Collect static files
echo "Collecting static files..."
python manage.py collectstatic --no-input
# Ensure the data directory exists
# I use the data directory to store files such as logs
mkdir -p data
# Start gunicorn
echo "Starting server..."
gunicorn core.wsgi:application --forwarded-allow-ips="*" --bind 0.0.0.0:8000
Worker entry point
#!/bin/sh
# Wait until the backend directory is created
until cd /app/backend
do
echo "Waiting for server volume..."
done
# run a worker
# I like having only one task per worker but you can change it
# by increasing the concurrency
echo "Starting celery worker..."
celery -A core worker -l info --concurrency 1 -E
Beat entry point
#!/bin/sh
# Wait until the server volume is available
until cd /app/backend
do
echo "Waiting for server volume..."
done
# run celery beat
echo "Starting celery beat..."
celery -A core beat -l info
Nginx image
This container serves the application.
I create a custom nginx image that includes my certificates and configuration, so I don't have to copy them over to the server.
Note: I don't use certbot, as I find it more straightfoward to generate the certificates from cloudflare and just store them in the custom image
This means that the image should be secure in a private registry with authentication, otherwise you risk security of your web app.
Here's the Dockerfile for it:
FROM nginx:stable-bullseye
# Export ports 80 and 443
EXPOSE 80
EXPOSE 443
# Copy the nginx configuration files to the image
COPY ./conf/default.conf /etc/nginx/conf.d/default.conf
COPY ./conf/prod.conf /etc/nginx/conf.d/prod.conf
COPY ./conf/staging.conf /etc/nginx/conf.d/staging.conf
# Copy the CloudFlare Origin CA certificate to the image
COPY ./certs/fullchain.pem /etc/nginx/certs/fullchain.pem
COPY ./certs/privkey.pem /etc/nginx/certs/privkey.pem
Redis image
I just use the default Redis image for this.
Just want to note that, because this is a single instance deployment, I like deploying Redis directly here as I find it's enough.
It is recommended, though, to spin up a Redis database somewhere more centralized.
Docker Compose
Environment Variables
Before I get into the gist of the Docker Compose file here are some environment variables I put in my .env file for deployment:
-
DOCKER_REGISTRY: My private, authentication enabled, docker registry where I upload the build images -
DJANGO_DOCKER_IMAGE: The name I give the django image -
NGINX_DOCKER_IMAGE: The name I give the NGINX image -
DOCKER_TAG: Usually the version I want to deploy, i.e.:1.5orlatest
The file
version: "3"
services:
redis:
container_name: redis
restart: unless-stopped
image: redis:7.2.0-alpine
expose:
- 6379
backend:
restart: unless-stopped
image: ${DOCKER_REGISTRY}/${DJANGO_DOCKER_IMAGE}:${DOCKER_TAG}
env_file:
- ./.env
entrypoint: /app/backend/entrypoint-django.sh
ports:
- 8000:8000
volumes:
- ./data:/app/backend/data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/healthcheck/"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s
worker:
restart: unless-stopped
image: ${DOCKER_REGISTRY}/${DJANGO_DOCKER_IMAGE}:${DOCKER_TAG}
env_file:
- ./.env
entrypoint: /app/backend/entrypoint-worker.sh
volumes:
- ./data:/app/backend/data
depends_on:
backend
condition: service_healthy
redis
condition: service_started
beat:
restart: unless-stopped
image: ${DOCKER_REGISTRY}/${DJANGO_DOCKER_IMAGE}:${DOCKER_TAG}
env_file:
- ./.env
entrypoint: /app/backend/entrypoint-beat.sh
volumes:
- ./data:/app/backend/data
depends_on:
backend
condition: service_healthy
redis
condition: service_started
nginx:
restart: unless-stopped
image: ${DOCKER_REGISTRY}/${NGINX_DOCKER_IMAGE}:${DOCKER_TAG}
ports:
- 80:80
- 443:443
depends_on:
backend
condition: service_healthy
As you can see, the compose file has the 5 services, Redis, django, celery worker, celery beat and NGINX.
Deploying
Building
First I build the images and push them to the registry. Before, I did this manually, now I use a GitHub action. You can learn more about this automation here.
Deploying with Docker Compose
Then I head to the server where I want to deploy this. Make sure that the .env file is updated and then just:
-
docker compose down: Spin the old instance down -
docker system prune -a -f: This makes sure I remove thelatestimage to force the download of the new one from the registry. -
docker compose up --scale worker=5 -d: Spin the new instance up That's it!
Top comments (2)
Nice setup! You could make it better by adding a containrrr/watchtower image. That way when you upload a new image to your registry watchtower will automatically download the new image a recreate your containers.
Nice one! Will investigate more, thanks for sharing!