DEV Community

Cover image for Messaging System with RabbitMQ, Celery, and Flask
augusthottie
augusthottie

Posted on

Messaging System with RabbitMQ, Celery, and Flask

#flask #celery #devops

Overview

This article walks you through setting up a messaging system using Flask, RabbitMQ, and Celery. The system handles asynchronous tasks like sending emails and logging timestamps. You can host the application on an AWS EC2 instance and use screen to keep it running persistently, while exposing it to the internet using ngrok.

GitHub Repo

For code details visit my repo ⬇️:

GitHub Repo

Table of Contents

Features

  • Send Emails: Use the ?sendmail parameter to send emails via SMTP.
  • Log Timestamps: Use the ?talktome parameter to log the current time.
  • View Logs: Access application logs via the /logs endpoint.
  • Asynchronous Tasks: Manage tasks asynchronously with RabbitMQ and Celery.

Technologies Used

  • Flask: A lightweight framework for building web applications.
  • Celery: An asynchronous task queue for managing background tasks.
  • RabbitMQ: A messaging broker that handles message queuing.
  • SMTP: Protocol used for sending emails.
  • ngrok: A tool to expose your local server to the internet.

Installation

Prerequisites

  • Python 3.7 or higher
  • pip (Python package installer)
  • RabbitMQ server installed and running

Step 1: Clone the Repository

First, clone the repository and navigate to the project directory:

git clone https://github.com/AugustHottie/devops-stage3.git
cd devops-stage3
Enter fullscreen mode Exit fullscreen mode

Step 2: Create a Virtual Environment

Create and activate a virtual environment:

python3 -m venv venv
source venv/bin/activate  # On Windows use `venv\Scripts\activate`
Enter fullscreen mode Exit fullscreen mode

Step 3: Install Dependencies

Install the necessary packages:

pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

Configuration

  1. Set Up Environment Variables

Create a .env file or set the variables directly in your shell:

   MAIL_ADDRESS=your-email@gmail.com
   APP_PASSWORD=your-google-app-password
   LOG_FILE_PATH=/var/log/messaging_system.log
Enter fullscreen mode Exit fullscreen mode
  1. Ensure RabbitMQ is Running

Start the RabbitMQ server:

   sudo systemctl start rabbitmq-server
Enter fullscreen mode Exit fullscreen mode

Running the Application

Step 1: Start the Celery Worker

Open a new terminal, activate your virtual environment, and start the Celery worker:

celery -A app.celery worker --loglevel=info
Enter fullscreen mode Exit fullscreen mode

Step 2: Start the Flask Application

In another terminal window, run the Flask application:

python app.py
Enter fullscreen mode Exit fullscreen mode

Endpoints

Endpoint Description Example Usage
/ Main route to send emails or log time http://localhost:8000/?sendmail=your_email@example.com
/ Main route to log the current time http://localhost:8000/?talktome
/logs View application logs http://localhost:8000/logs

Logging

Logs are saved to the path specified by LOG_FILE_PATH. Make sure the application has permission to write to this location.

Using ngrok

To expose your local application to the internet, follow these steps:

  1. Download and Install ngrok 
   wget https://bin.equinox.io/c/4b0e5f0d1d6e/ngrok-stable-linux-amd64.zip
   unzip ngrok-stable-linux-amd64.zip
Enter fullscreen mode Exit fullscreen mode
  1. Add Your ngrok Authtoken

Sign up or log in to your ngrok account to get your authtoken. Once you have it, run:

   ./ngrok authtoken your_ngrok_auth_token
Enter fullscreen mode Exit fullscreen mode

Replace your_ngrok_auth_token with the token you received from the ngrok dashboard.

  1. Expose Your Flask Application 
   ./ngrok http 8000
Enter fullscreen mode Exit fullscreen mode
  1. Use the Provided ngrok URL

Use the URL provided by ngrok to access your application externally.

Hosting on AWS EC2

Step 1: Launch an EC2 Instance

  1. Log in to AWS Management Console and navigate to EC2.
  2. Launch a new EC2 instance with your preferred configuration.
  3. Configure security groups to allow HTTP (port 80) and custom TCP (port 8000) traffic.

Step 2: Connect to Your EC2 Instance

Use SSH to connect to your EC2 instance:

ssh -i /path/to/your-key.pem ec2-user@your-ec2-public-dns
Enter fullscreen mode Exit fullscreen mode

Step 3: Install Dependencies on EC2

  1. Update the package list and install necessary packages: 
   sudo yum update -y
   sudo yum install -y python3-pip
Enter fullscreen mode Exit fullscreen mode
  1. Clone the repository, create a virtual environment, and install dependencies: 
   git clone https://github.com/AugustHottie/devops-stage3.git
   cd devops-stage3
   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode
  1. Ensure RabbitMQ is installed and running: 
   sudo yum install -y rabbitmq-server
   sudo systemctl start rabbitmq-server
Enter fullscreen mode Exit fullscreen mode

Step 4: Set Up screen to Persist the Application

  1. Install screen if it is not already installed: 
   sudo yum install -y screen
Enter fullscreen mode Exit fullscreen mode
  1. Start a new screen session: 
   screen -S myapp
Enter fullscreen mode Exit fullscreen mode
  1. Run the Flask application within the screen session: 
   python app.py
Enter fullscreen mode Exit fullscreen mode

  1. Detach from the screen session by pressing Ctrl+A, then D.

  2. To reattach to the screen session:

   screen -r myapp
Enter fullscreen mode Exit fullscreen mode

Step 5: Set Up ngrok on EC2

  1. Download and install ngrok: 
   wget https://bin.equinox.io/c/4b0e5f0d1d6e/ngrok-stable-linux-amd64.zip
   unzip ngrok-stable-linux-amd64.zip
Enter fullscreen mode Exit fullscreen mode
  1. Add your ngrok authtoken: 
   ./ngrok authtoken your_ngrok_auth_token
Enter fullscreen mode Exit fullscreen mode

Replace your_ngrok_auth_token with the token you received from the ngrok dashboard.

  1. Expose your Flask application using ngrok: 
   ./ngrok http 8000
Enter fullscreen mode Exit fullscreen mode
  1. Use the provided ngrok URL to access your application externally.

This guide should help you get your messaging system up and running both locally and on an AWS EC2 instance. Feel free to share your feedback or suggestions in the comment section! 🚀

Top comments (2)

Collapse
 
whattheportal profile image
WTP | WhatThePortal.com • • Edited

Thanks for spelling all this out!

There are a lot of moving pieces in an event driven architecture, with each one being seemingly critical to the operation of the whole.

When switching to this type of system design ourselves (from a traditional REST-based set of patterns), we noticed it really beefed up the importance of teaching the team how to create quick and efficient pull requests that wouldn't crash the whole thing.

Thanks for the writeup!

Collapse
 
augusthottie profile image
augusthottie •

Thanks for the great feedback!

Read next

suin profile image

Sharing Secrets Between Kubernetes Clusters Using external-secrets PushSecret

suin -

mrcaption49 profile image

SDLC | Waterfall model | Agile | DevOps

Pranav Bakare -

hana_sato profile image

Talend vs. Apache Kafka: Which Data Tool Drives Better Business Insights?

Hana Sato -

ekirigwe profile image

Securing CI/CD Pipelines: GitHub Actions vs Jenkins

Samuel Ekirigwe -