DJANGO CELERY SEND EMAILS TO ALL USERS

In this guide, we're going to send emails to all our users on our Django web application asynchronously. Imagine a very large application where we have thousands of users and we want to email them the new and latest changes on our platform, probably some newsletter or something. We can't afford to allow the traditional Django to handle this for us as this will take a lot of time to complete this. Time waiting for our email service to send all these emails can be used to do something else by the user and it is not a very good experience for our users. So in this case, we need some sort of performance task that can improve our user's activity.

Celery does the work for us. It enables our Django application to complete various task queues for multiple workers.

For this guide, we'll use Redis as our message broker to get the performance.

The first thing first is to install Django, Celery and Redis with the commands below

pip install Django==3.2.10
pip install celery==4.4.2
pip install redis==4.3.4

So let's start a new Django project named "my_project". Once our project is created, we can now change the directory into this newly created project and run our server

django-admin startproject my_project
cd my_project
python manage.py runserver

If our server is running just fine, then we're on the right track and now need to start setting up celery for our project.

In my_project app, we need to create a new file named "celery.py". In this file, this is where all our celery configurations will be located. In this file, let's add the code below;

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

# setting the Django settings module.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'my_project.settings')
app = Celery('ecommerce')
app.config_from_object('django.conf:settings', namespace='CELERY')

# Looks up for task modules in Django applications and loads them
app.autodiscover_tasks()

This last line of code tells Celery to automatically discover files named tasks.py in all registered Django apps.

Now in the same main project app's folder where we have our celery.py and settings file, we need to update our init.py file to tell Celery to automatically start whenever Django starts. Update our init.py file with the code below;

from .celery import app as celery_app

__all__ = ('celery_app',)

Next, we need to specify Redis as our broker for celery in our settings file. Add the line of code below to our settings.py

# CELERY SETTINGS
CELERY_BROKER_URL = 'redis://127.0.0.1:6379'

Now we have all our celery configurations in place, our next steps will require us to create a view where we can perform the necessary actions and further integrate celery to make this asynchronous.

Let's create a new app name public;

python manage.py startapp public

Once created, add it to our installed apps in the settings file;

INSTALLED_APPS = [
    ...
    'public',
]

We can now migrate our database with the comma nd below;

python manage.py migrate

Next is to create our home view in the public app to handle all things for us, add the code below

from django.shortcuts import render
from django.contrib.auth.models import User


def home_view(request):
    users = User.objects.all()
    context = {
        'users': users
    }
    return render(request, 'public/home.html', context)

We now have our simple view which will simply render our HTML page for us with a context that lists all our users. This will help us in our home view templates to know the number of users on our platform.

Our next step is to create our urls.py file in our public app, and add the code below to our urls.py file.

from django.urls import path
from .views import home_view

app_name = "public"

urlpatterns = [
    path('', home_view, name='home-view'),
]

We now have our views and it's URLs, lastly, we need to include the public's URLs in the main project URLs file so that Django will be able to render it.

Now in our main project urls.py file, we can update it with the code below;

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('public.urls', namespace='public'))
]

Now we have this in place, we can now run our server. We'll see an error in the browser which says
TemplateDoesNotExist at /public/home/html

If you get this error, it shows you're doing things right. The last thing to fix this error is to go to our settings.py file and tell Django how to locate our templates files

TEMPLATES = [
    {
        ...
        'DIRS': [BASE_DIR / 'templates'],
      ...
    },
]

With this in place, we can now create a templates folder in our main project directory and proceed to create a public folder in the templates folder and finally create a home.html file in the public folder.
templates/public/home.html

Now we need to update our home.html file with the code below;

<!doctype html>
<html lang="en">

<head>
  <!-- Required meta tags -->
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">

  <!-- Bootstrap CSS -->
  <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet"
    integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous">

  <title>Hello, world!</title>

</head>

<body>

  <style>
    .centralize-div {
      display: flex !important;
      justify-content: center !important;
    }
  </style>

  <nav class="navbar navbar-expand-lg navbar-light bg-light">
    <div class="container-fluid">
      <a class="navbar-brand" href="#">Django Users</a>
      <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarSupportedContent"
        aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
        <span class="navbar-toggler-icon"></span>
      </button>
      <div class="collapse navbar-collapse" id="navbarSupportedContent">
        <ul class="navbar-nav me-auto mb-2 mb-lg-0">
          <li class="nav-item">
            <a class="nav-link active" aria-current="page" href="#">Posts</a>
          </li>
        </ul>
      </div>
    </div>
  </nav>

  <div class="container py-5">
    <div class="row">
        <h1>There are {{ users.count }} users</h1>
    </div>
  </div>

  <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js"
    integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" crossorigin="anonymous">
    </script>

</body>
</html>

In the code above, we are only using the Django templating language to tell Django how many users we have on our Django project. Currently, we have zero, so we have to create a super user so we can add more and more users to our platform.

python manage.py createsuperuser

Now we have some users on our platform, we can see that when we visit the home page we see a zero change to the number of users we just created in the admin panel.

We now have our basic Django setup, it's now time to work on our celery function. In our case, I want to keep things simple so we'll first write the most basic email function and include it on our home page so that when the page loads it will run the function. Once this works fine, then we can proceed to make it asynchronous with celery.

Now create a new file "tasks" in our public app and add the code below;

from django.contrib.auth.models import User
from django.core.mail import send_mail

from celery import task
from celery.utils.log import get_task_logger


logger = get_task_logger(__name__)


# @task
def send_emails_to_users():
    users = User.objects.all()
    users_emails = []
    for user in users:
        users_emails.append(user.email)

    sending_mail = send_mail(
        'Subject here',
        'Here is the message.',
        'from@example.com',
        [users_emails],
        fail_silently=False,
    )
    print("Voila, Email Sent to " + str(len(users)) + " users")
    return sending_mail

In our code above, we're importing the Django User model and send_mail for sending emails with Django. We're also import task which is what celery uses to send our function to queue to handle our function asynchronously. Even if our function just doesn't run maybe because of some failure, celery tasks will help us deliver our function to another worker to run the function.

Next is our function that sends the email to all our users. First we set our users to be all the users on the platform then we have an empty list of users_emails. Our for loop helps us to get all the emails of all users and we now append these emails into our empty user_emails list.

Last block sends emails to all the users on our platform. Once you have all these in place, the last steps is to add this line of code below in our settings file for our email setup. For the purpose of this tutorial, we'll print our emails to the console and not sending actual emails to our user's email addresses

# Email Settings
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

Alright, now we need to add our email function in our views.py file so our function will be called whenever we load our homepage. Update our views.py with the code below;

from django.shortcuts import render
from django.contrib.auth.models import User
from .tasks import send_emails_to_users


def home_view(request):
    users = User.objects.all()
    context = {
        'users': users
    }
    send_emails_to_users()
    return render(request, 'public/home.html', context)

All good, we now have everything working just fine. Run the server and monitor the command line for the outputs you'll get.

In our console, we can see it prints the email and the email message being sent and also the print statement we added to our email function.

Now, this is working as expected, but it is not using celery and it is not asynchronous. For us to make it asynchronous, we now need to uncomment "@task" in our email function in tasks.py. This tells celery to handle our function. Our final code will look like this;

from django.contrib.auth.models import User
from django.core.mail import send_mail

from celery import task
from celery.utils.log import get_task_logger


logger = get_task_logger(__name__)


@task
def send_emails_to_users():
    users = User.objects.all()
    users_emails = []
    for user in users:
        users_emails.append(user.email)

    sending_mail = send_mail(
        'Subject here',
        'Here is the message.',
        'from@example.com',
        [users_emails],
        fail_silently=False,
    )
    print("Voila, Email Sent to " + str(len(users)) + " users")
    return sending_mail

Next is to update our views.py by adding the .delay which is a celery method for calling tasks and is very convenient. Our final views.py should look like this;

from django.shortcuts import render
from django.contrib.auth.models import User
from .tasks import send_emails_to_users


def home_view(request):
    users = User.objects.all()
    context = {
        'users': users
    }
    send_emails_to_users.delay()
    return render(request, 'public/home.html', context)

Now run our server, you'll notice that our server still works perfectly fine but not with celery and our email function doesn't work. The very last step is to open another command line in the same project folder. This will be specifically for celery worker, use the command below;

celery -A my_project worker --pool=solo -l info

We're all set, now in the celery worker's console, you'll see the output of the previous result of our initial request to the home view webpage.

Thank you all and this is all about this celery guide, we can make things better by adding an email service to send actual emails to our user's email addresses. If you have any questions, feel free to reach out to me, the final code repository for this project is in the GitHub repo below;

https://github.com/Arthurobo/django-celery-send-emails-to-all-users