Deploying A Django App: EC2 vs App Runner with External Celery

Wait a minute...

We have all encountered this situation where we are busy trying to go to production, but a lot of factors account for the choice of your platform of deployment. Emmmm YES, we will go with AWS. Usually after sticking to a platform, we can now rely on some factors such as: architecture, cost, reliability, scalability, availability and feasibility. Guess what!!! This will not be about reliability, scalability, availability and feasibility since AWS is trusted for all those. In this tutorial we will identify the ups and downs of some architecture for your Django App.

Before we proceed let's understand a few prerequisites to perfectly understand what is going on.

:) All the code involved in this tutorial will be available as opensource. Feel free to put your footprint into it.

Prerequisites

Before moving ahead, you are required to:

• Have an AWS account
• Have some Django knowledge
• Understand what queueing, tasks, brokers are

What is Caching and Why do we Cache

Caching is a technique used to temporarily store frequently accessed data in a fast-access location, reducing the time it takes to retrieve this data. In AWS, caching improves application performance and scalability by minimizing the load on primary databases and APIs, thereby speeding up response times for end-users.

We cache to enhance efficiency, reduce latency, and lower costs. By storing data closer to the application, caching decreases the frequency of database queries, network traffic, and computational load. This results in faster data retrieval, improved user experience, and optimized resource usage, which is crucial for high-traffic applications.

Let's be warming up

1. EC2:
   From its full meaning Elastic Compute Engine, EC2 are web servers found in AWS datacenters. In others words, EC2 are virtual which you can get from AWS. With all functionalities available you can get one at a very cheap monthly rate under a "pay-as-you-go plan".

2. AWS App Runner:
   This is a fully managed service that simplifies running and scaling web applications and APIs, allowing developers to quickly deploy from code repositories or container images without infrastructure management.

3. Celery and Django Celery:
   Celery is an open-source distributed task queue for real-time processing in Python. Django Celery integrates Celery with the Django framework, enabling asynchronous task execution, periodic tasks, and background job management within Django applications. The use case of this technology varies. It can be communication services (SMS, emails), Scheduled Jobs (Crons), and background data processing tasks, such as data aggregation, machine learning model training, or file processing.

4. Amazon RDS (Relational Database Service):
   It is a managed database service that simplifies setting up, operating, and scaling relational databases in the cloud. It supports various database engines like MySQL, PostgreSQL, Oracle, and SQL Server, providing automated backups, patching, and high availability, freeing users from database administration tasks.

Comparing the EC2 and App Runner in this context

Architectures

Let's study how the app is structured and how the deployment setup would behave.

1. Deployment setup with AWS App Runner (ECR)
   
   We push our code to GitHub, triggering a CodePipeline workflow. CodePipeline uses CodeBuild to create Docker images stored in Elastic Container Registry (ECR) for versioning releases. This tutorial skips Virtual Private Cloud (VPC) configuration. We ensure application health by constantly monitoring logs using CloudWatch. And a bonus is the quick configuration of the project to use Postgres provide by the AWS RDS and S3 for static files.

2. Deployment with AWS EC2 Instance
   
   Using a similar process, omitting versioning and ECR, we push our code to GitHub, triggering CodePipeline, which uses CodeBuild to create Docker images stored in ECR for versioning. EC2 instances pull these images to deploy the application within a VPC, making it accessible to end users. The application interacts with RDS for data storage and S3 for static files, monitored by CloudWatch. Optionally we can add an SSL configuration into this instance with options like certbot.

Price Comparison Table

Here’s a hypothetical price comparison between EC2 and App Runner based on typical usage scenarios:

Ease of Management

Let's have a quick summary about how managing these two resources go.

Scalability

Deployment Speed

Customization and Control

Security

Project Setup

Given that the comparison of our project does not rely on the project setup itself. We will have a basic Django application with a celery configuration from AWS.
We will go with a basic project using Django.

Installing Dependencies and Project Creation:

The commands should be run in the order below:

# Project directory creation
mkdir MySchedular && cd MySchedular

# Creating an isolated space for the project dependencies
python -m venv venv && source venv/bin/activate

# Dependencies installation
pip install django celery redis python_dotenv

# Creating project and app
django-admin startproject my_schedular . && python manage.py startapp crons

# Let's add a few files to the project skeleton
touch my_schedular/celery.py crons/urls.py crons/tasks.py

At this point in time we can check our project skeleton with this:

tree -I "venv|__pycache__" .

And we should have this one at the moment

    .
    ├── crons
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── migrations
    │   │   └── __init__.py
    │   ├── models.py
+   │   ├── tasks.py
    │   ├── tests.py
+   │   ├── urls.py
    │   └── views.py
    ├── manage.py
    └── my_schedular
        ├── __init__.py
        ├── asgi.py
+       ├── celery.py
        ├── settings.py
        ├── urls.py
        └── wsgi.py

    3 directories, 16 files

Code and Logic

We can proceed now by adding a couple of lines for the logic of out app and covering another milestone for this project.
1- Setting up the celery

# my_schedular/celery.py

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')

app = Celery('myproject')

app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print(f'Request: {self.request!r}')

2- Let's overwrite the celery variables to set our broker

# my_schedular/settings.py

CELERY_BROKER_URL = os.getenv('CELERY_BROKER_URL ')
CELERY_RESULT_BACKEND = os.getenv('CELERY_RESULT_BACKEND')

3- Update init.py to ensure the app is loaded when Django starts:

# my_schedular/__init__.py
from __future__ import absolute_import, unicode_literals

from .celery import app as celery_app

__all__ = ('celery_app',)

4- We create our task

# crons/tasks.py
from celery import shared_task
import time

@shared_task
def add(x, y):
    time.sleep(10)
    return x + y

5- Let's add our view now, just a simple one with a simple Json response.

# crons/views.py
from django.http import JsonResponse
from crons.tasks import add

def index(request):
    return JsonResponse({"message": "Hello world, your Django App is Running"})

def add_view(request):
    result = add.delay(4, 6)
    return JsonResponse({'task_id': result.id})

6- We cannot have a view, without an endpoint to make it possible to access it

# crons/urls.py
from django.urls import path

from crons.views import add_view, index

urlpatterns = [
    path('', index, name='index'),
    path('add/', add_view, name='add'),
]

7- Adding our apps urls to the general urls.py of the whole project.

# my_schedular/urls.py
from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path('admin/', admin.site.urls),
    path('/', include('crons.urls')),
]

Adding Environment Variables:

# .env
SECRET_KEY=
DEBUG=
CELERY_BROKER_URL=
CELERY_RESULT_BACKEND=

After proper follow up of all these steps, we have this output:

AWS Environment Setup

Since we are shipping to AWS We need to configure a few resource to

Creating a new VPC (Virtual Private Cloud)

We create an isolated environment and a network for a secure access and communication between our resources.

Creating Security Groups

We create a security group under the previously made VPC and together add inbound and outbound rules to the TCP port 6379 (the Redis Port).

Creating the RedisOSS from ElasticCache

Basically, AWS Elastic Cache offers us two varieties when it comes to caching, namely: RedisOSS and memCache. RedisOSS offers advanced data structures and persistence features, while Memcached is simpler, focusing on high-speed caching of key-value pairs. Redis also supports replication and clustering, unlike Memcached. Back to business, back to Redis.

Elastic Container Registry (ECR) Setup

The Creation of an ECR image will be very simple and straight forward.

ONE: Updates to deploy the App Runner

Follow the steps below to have your app runner running.



Here we need to be very technical. A VPC is a secured network where most of our resources lie, since an App runner is not found into a VPC, we will need to provide a secured means for communication between those resources.

Credentials user credentials

For this tutorial we will need an authorization to connect our workflow to our ECR. Then we add the AmazonEC2ContainerRegistryFullAccess permission policy so it can push the image to our AWS ECR.

Results

When all is done we have this tree structure.

You can have the whole code base for this tutorial on My GitHub.

TWO: Deploying to an EC2

We will go with one the easiest EC2 to setup and the one having a free tier, an ubuntu EC2 instance. And The same code base that was used above is the same we are using here.

Creating an EC2

![EC2 1]https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rk8waijxkthu1ule91fn.png)

Alternatively, we can setup the security group separately.

Setting up the EC2

Run this script to install necessary dependencies

#!/bin/bash

# Update the package list and upgrade existing packages
sudo apt-get update
sudo apt-get upgrade -y

# Install Python3, pip, and other essentials
sudo apt-get install -y python3-pip python3-dev libpq-dev nginx curl

# Install Redis
sudo apt-get install -y redis-server

# Start and enable Redis
sudo systemctl start redis.service
sudo systemctl enable redis.service

# Install Supervisor
sudo apt-get install -y supervisor

# Install virtualenv
sudo apt install python3-virtualenv

# Setup your Django project directory (adjust the path as needed)
cd ~/aws-django-redis

# Create a virtual environment
virtualenv venv

# Activate the virtual environment
source venv/bin/activate

# Install Gunicorn and other requirements
pip install gunicorn
pip install -r requirements.txt

# Create directories for logs if they don't already exist
sudo mkdir -p /var/log/aws-django-redis
sudo chown -R ubuntu:ubuntu /var/log/aws-django-redis

# Supervisor Configuration for Gunicorn
echo "[program:aws-django-redis]
command=$(pwd)/venv/bin/gunicorn --workers 3 --bind 0.0.0.0:8000 my_schedular.wsgi:application
directory=$(pwd)
autostart=true
autorestart=true
stderr_logfile=/var/log/aws-django-redis/gunicorn.err.log
stdout_logfile=/var/log/aws-django-redis/gunicorn.out.log
user=ubuntu
" | sudo tee /etc/supervisor/conf.d/aws-django-redis.conf

# Supervisor Configuration for Celery
echo "[program:celery]
command=$(pwd)/venv/bin/celery -A my_schedular worker --loglevel=info
directory=$(pwd)
autostart=true
autorestart=true
stderr_logfile=/var/log/aws-django-redis/celery.err.log
stdout_logfile=/var/log/aws-django-redis/celery.out.log
user=ubuntu
" | sudo tee /etc/supervisor/conf.d/celery.conf

# Reread and update Supervisor
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl restart all

# Set up Nginx to proxy to Gunicorn
echo "server {
    listen 80;
    server_name <your_vm_ip>;

    location / {
        proxy_pass http://127.0.01:8000;
        proxy_set_header Host \$host;
        proxy_set_header X-Real-IP \$remote_addr;
        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto \$scheme;
    }

    error_log  /var/log/nginx/aws-django-redis_error.log;
    access_log /var/log/nginx/aws-django-redis_access.log;
}" | sudo tee /etc/nginx/sites-available/aws-django-redis

# Enable the Nginx site configuration
sudo ln -s /etc/nginx/sites-available/aws-django-redis /etc/nginx/sites-enabled/
sudo rm /etc/nginx/sites-enabled/default

# Test Nginx configuration and restart Nginx
sudo nginx -t
sudo systemctl restart nginx

Results

This setup is available on GitHub on the dev branch, have a look and open a PR.

Pricing and Setup Comparison Table

Cost Summary

• Setup Using AWS Free Tier: Primarily free if staying within the free tier limits. Potential costs may arise if usage exceeds free tier allowances.
• Setup Using All Paid AWS Services: Estimated around $41.34 per month, assuming continuous operation of one t2.micro instance for EC2, Elasticache, and RDS, with additional costs for data transfer and storage.

Note: Prices are approximate and can vary based on region and specific AWS pricing changes. Always check the current AWS Pricing page to get the most accurate cost estimates for your specific requirements.

Analysis

• Self-Managed on EC2: This approach is cost-effective, especially with the use of the AWS free tier. It requires more setup and manual maintenance but provides full control over the environment. Ideal for smaller scale or lower budget projects.
• Fully Managed AWS Services: While this increases operational costs, it reduces the workload related to infrastructure management, scaling, and maintenance. It’s suitable for larger applications or when operational simplicity and scaling are priorities.

Summary

Nahhhhhhhhhh!!! Unfortunately, there is no summary for this one. Yes, go back up for a better understanding.

Conclusion

The learning path is long and might seems difficult, but one resource at a time, continuously appending knowledge leads us to meet our objectives and goal.