DEV Community

Afeez Oluwashina Adeboye
Afeez Oluwashina Adeboye

Posted on • Edited on

Complete Guide to Containerizing and Deploying a Full-Stack Application with Docker Compose, Nginx Manager, and Monitoring Tools

#devops #monitoring

Containerizing a Full-Stack Application with Advanced Monitoring

Table of Contents

A. Introduction
B. Prerequisites
C. Project Setup
D. Backend Configuration
E. Frontend Configuration
F. Docker Setup
G. Docker Compose Configuration
H. Reverse Proxy & SSL Setup
I. Database Management
F. Setting up the monitoring stack

Introduction

In this project, we will containerize and deploy a multi-tier web application using Docker, Docker Compose, and reverse proxy configurations. The application consists of a React frontend that interacts with a FastAPI backend, which is backed by a PostgreSQL database. To ensure smooth monitoring and management of the application's performance, we will integrate a Monitoring Stack that includes Prometheus for metrics collection, Grafana for data visualization, Loki for log aggregation, and cAdvisor for container performance monitoring.

The overall architecture will use Docker Compose for orchestration, ensuring that all services are interconnected and can communicate seamlessly. The services will be routed through a reverse proxy, either Traefik or Nginx Proxy Manager, to ensure proper URL routing, security, and access control for both the application stack and the monitoring stack.

Prerequisites

  • Linux-based operating system (Ubuntu, CentOS, etc.)
  • Docker and Docker Compose installed
  • Cloud provider account (AWS, Google Cloud, or Azure) for deployment
  • Domain name
  • Basic knowledge of:
    • Docker & Docker Compose
    • React & FastAPI
    • Prometheus, Grafana, Loki, and cAdvisor

Project Setup

A. Clone the repository:

git clone https://github.com/The-DevOps-Dojo/cv-challenge01.git
cd cv-challenge01
Enter fullscreen mode Exit fullscreen mode

Backend Configuration

A. Navigate to the backend directory:

cd backend
Enter fullscreen mode Exit fullscreen mode

B. Install Poetry (dependency manager):

  • For macOS/Linux:

     curl -sSL https://install.python-poetry.org | python3 -

  • For Windows: Follow Poetry's documentation

Poetry Installation

C. Verify Poetry installation:

poetry --version
Enter fullscreen mode Exit fullscreen mode

Poetry Version Check

D. Install project dependencies:

poetry install
Enter fullscreen mode Exit fullscreen mode

PostgreSQL Setup

A. Install PostgreSQL:

sudo apt update
sudo apt install postgresql postgresql-contrib
Enter fullscreen mode Exit fullscreen mode

B. Switch to PostgreSQL user:

sudo -i -u postgres
psql
Enter fullscreen mode Exit fullscreen mode

C. Create database and user:

CREATE USER app WITH PASSWORD 'changethis123';
CREATE DATABASE app;
\c app
GRANT ALL PRIVILEGES ON DATABASE app TO app;
GRANT ALL PRIVILEGES ON SCHEMA public TO app;
\q
exit
Enter fullscreen mode Exit fullscreen mode

D. Configure environment variables:
Create/edit .env file in the backend directory:

POSTGRES_SERVER=localhost
POSTGRES_PORT=5432
POSTGRES_DB=app
POSTGRES_USER=app
POSTGRES_PASSWORD=changethis123
Enter fullscreen mode Exit fullscreen mode

Environment Configuration

E. Initialize database:

poetry run bash ./prestart.sh
Enter fullscreen mode Exit fullscreen mode

F. Start backend service:

poetry run uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
Enter fullscreen mode Exit fullscreen mode

Frontend Configuration

A. In a new terminal, navigate to frontend:

cd frontend
Enter fullscreen mode Exit fullscreen mode

B. Install Node.js and dependencies:

sudo apt update
sudo apt install nodejs npm
npm install
Enter fullscreen mode Exit fullscreen mode

C. Start frontend server:

npm run dev -- --host
Enter fullscreen mode Exit fullscreen mode

When attempting to log in, you might encounter a network error:

Network Error

Frontend Configuration Updates

A. Update API URL in frontend .env:

VITE_API_URL=http://<your_server_IP>:8000
Enter fullscreen mode Exit fullscreen mode

After updating the frontend configuration, you may encounter a CORS error:

CORS Error

B. Update CORS settings in backend .env:

BACKEND_CORS_ORIGINS="http://localhost,http://localhost:5173,https://localhost,https://localhost:5173,http://<your_server_IP>:5173"
Enter fullscreen mode Exit fullscreen mode

C. Use these default login credentials:

  • Username: chanllenge@devopsdojo.com
  • Password: devopsdojo57

After successful configuration, you should see:

Successful Login

Accessing Swagger Documentation

Access the Swagger API documentation at http://<your_server_IP>:8000/docs:

Swagger Documentation

Docker Setup

Backend Dockerfile

Create a Dockerfile in the backend directory:

FROM tiangolo/uvicorn-gunicorn-fastapi:python3.10
WORKDIR /app/
RUN curl -sSL https://install.python-poetry.org | POETRY_HOME=/opt/poetry python && \
    cd /usr/local/bin && \
    ln -s /opt/poetry/bin/poetry && \
    poetry config virtualenvs.create false
COPY ./pyproject.toml ./poetry.lock* /app/
RUN poetry install --no-root --only main
ENV PYTHONPATH=/app
COPY ./alembic.ini /app/
COPY ./prestart.sh /app/
COPY ./app /app/app

# Ensure prestart.sh is executable
RUN chmod +x /app/prestart.sh

# Override the CMD to run the prestart.sh script and then start Uvicorn
CMD ["/bin/bash", "-c", "/app/prestart.sh && uvicorn app.main:app --host 0.0.0.0 --port 8000"]
Enter fullscreen mode Exit fullscreen mode

Frontend Dockerfile

Create a Dockerfile in the frontend directory:

# Use the latest official Node.js image as a base
FROM node:latest

# Set the working directory
WORKDIR /app

# Copy the application files
COPY . .

# Install dependencies
RUN npm install

# Expose the port the development server runs on
EXPOSE 5173

# Run the development server
CMD ["npm", "run", "dev", "--", "--host"]
Enter fullscreen mode Exit fullscreen mode

Docker Compose Configuration

A. Create a docker-compose.yml file in the project root:

services:
  backend:
    build:
      context: ./backend
    container_name: backend_fastapi_app
    ports:
      - "8000:8000"
    depends_on:
      - db
    env_file:
      - ./backend/.env

  frontend:
    build:
      context: ./frontend
    container_name: frontend_nodejs_app
    ports:
      - "5173:5173"
    env_file:
      - ./frontend/.env

  db:
    image: postgres:latest
    container_name: postgres_db
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    env_file:
      - ./backend/.env

  adminer:
    image: adminer
    container_name: adminer-service
    ports:
      - "8080:8080"

  proxy:
    image: jc21/nginx-proxy-manager:latest
    container_name: nginx_proxy_manager
    ports:
      - "80:80"
      - "443:443"
      - "81:81"
    environment:
      DB_SQLITE_FILE: "/data/database.sqlite"
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
      - backend
      - frontend
      - adminer

volumes:
  postgres_data:
  data:
  letsencrypt:
Enter fullscreen mode Exit fullscreen mode

B. Start the services:

# Build and start all services
docker compose up --build -d

# Check service status
docker compose ps

# View logs
docker compose logs -f
Enter fullscreen mode Exit fullscreen mode

Reverse Proxy and SSL Setup

A. Access Nginx Proxy Manager

  • URL: http://<your-server-ip>:81
  • Default credentials:
    • Email: admin@example.com
    • Password: changeme

Nginx Proxy Manager Login

B. Configure Proxy Hosts for Services

  • Add Frontend Proxy Host
    • Domain: your_domain.com
    • Forward Hostname: frontend
    • Forward Port: 5173
    • Options:
      • Enable Block Common Exploits

Proxy Host Configuration

  • SSL:
    • Request a Let's Encrypt certificate.
    • Enable Force SSL.

SSL PROXY MANAGER

C. Configure Advanced Proxy Settings

Go to the Advanced tab for the backend service and add the following routing rules:

   location /api {
       proxy_pass http://backend:8000;
       proxy_set_header Host $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
   }

   location /docs {
       proxy_pass http://backend:8000;
       proxy_set_header Host $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
   }
Enter fullscreen mode Exit fullscreen mode

Advanced Configuration

  • Add Nginx Manager Proxy Host (API)

    • Domain: proxy.your_domain.com
    • Forward Hostname: proxy
    • Forward Port: 81
    • SSL: Follow the same steps as above.

  • Add Adminer Proxy Host

    • Domain: db.your_domain.com
    • Forward Hostname: db
    • Forward Port: 8080
    • SSL: Follow the same steps as above.

  • Redirect www to Non-www:

    • Navigate to Proxy Hosts in Nginx Proxy Manager.
    • Add a new Proxy Host for www.your_domain.com.
    • Forward Hostname: your_domain.com
    • Forward Port: 443
    • Enable Force SSL.

Redirect Configuration

D. Verify Service Accessibility

After completing the above configurations:

  • Frontend: https://your_domain.com
  • Backend API: https://api.your_domain.com
  • Adminer: https://db.your_domain.com

Database Management

A. Access Adminer

  1. Navigate to https://db.your_domain.com.
  2. Login with the credentials from the backend's .env file:
    • System: PostgreSQL
    • Server: db

Adminer Login Page

B. Manage Database

  • Use Adminer to perform tasks like creating tables, managing records, or running SQL queries.
  • Example Usage:
    • List All Tables: Access the left navigation menu in Adminer after logging in.
    • Run Queries: Use the SQL tab to execute custom queries.

db management

By following these steps, you have a structured reverse proxy and database management setup. The dashboard ensures www requests are redirected seamlessly, and all services are accessible via specific subdomains.

Setting Up a Complete Monitoring Stack for Your Containerized Application

NOTE: Please update the application compose file with a shared network. This is to make all services communicate internally with each other as shown below

services:
  backend:
    build:
      context: ./backend
    container_name: backend_fastapi_app
    ports:
      - "8000:8000"
    depends_on:
      - db
    env_file:
      - ./backend/.env
    networks:
      - shared-network

  frontend:
    build:
      context: ./frontend
    container_name: frontend_nodejs_app
    ports:
      - "5173:5173"
    env_file:
      - ./frontend/.env
    networks:
      - shared-network

  db:
    image: postgres:latest
    container_name: postgres_db
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    env_file:
      - ./backend/.env
    networks:
      - shared-network

  adminer:
    image: adminer
    container_name: adminer-service
    ports:
      - "8080:8080"
    networks:
      - shared-network

  proxy:
    image: jc21/nginx-proxy-manager:latest
    container_name: nginx_proxy_manager
    ports:
      - "80:80"
      - "443:443"
      - "81:81"
    environment:
      DB_SQLITE_FILE: "/data/database.sqlite"
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
      - backend
      - frontend
      - adminer
    networks:
      - shared-network

volumes:
  postgres_data:
  data:
  letsencrypt:

networks:
  shared-network:
    name: shared-network
Enter fullscreen mode Exit fullscreen mode

Now that we have our application running, let's add comprehensive monitoring capabilities. We'll set up Prometheus for metrics collection, Grafana for visualization, cAdvisor for container metrics, and Loki with Promtail for log aggregation.

Getting Started

First, create a new file called monitoring-compose.yml:

services:
  grafana:
    image: grafana/grafana
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SERVER_DOMAIN=your.domain
      - GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
      - GF_AUTH_PROXY_ENABLED=false
      - GF_SECURITY_ALLOW_EMBEDDING=true
      - GF_SECURITY_COOKIE_SAMESITE=none
      - GF_SECURITY_COOKIE_SECURE=true
      - GF_SESSION_PROVIDER=memory
      - GF_SESSION_PROVIDER_CONFIG=sessions
    ports:
      - "3000:3000"
    volumes:
      - grafana-storage:/var/lib/grafana
    networks:
      - shared-network

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.external-url=/prometheus'
      - '--web.route-prefix=/prometheus'
    ports:
      - "9090:9090"
    restart: unless-stopped
    volumes:
      - ./prometheus:/etc/prometheus 
      - prom_data:/prometheus
    depends_on:
      - cadvisor
    networks:
      - shared-network

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
      - "8083:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    networks:
      - shared-network

  loki:
    image: grafana/loki:3.0.0
    container_name: loki
    user: "10001"
    volumes:
      - ./loki-config.yaml:/mnt/config/loki-config.yaml
      - ./data:/tmp/loki
    ports:
      - "3100:3100"
    command: -config.file=/mnt/config/loki-config.yaml
    networks:
      - shared-network

  promtail:
    image: grafana/promtail:3.0.0
    container_name: promtail
    depends_on:
      - loki
    volumes:
      - ./promtail-config.yaml:/mnt/config/promtail-config.yaml
      - /var/log:/var/log
    command: -config.file=/mnt/config/promtail-config.yaml
    networks:
      - shared-network

volumes:
  grafana-storage:
  prom_data:

networks:
  shared-network:
    external: true
Enter fullscreen mode Exit fullscreen mode

Important Configuration Notes

Network Configuration

Remember the shared-network we created earlier? Our monitoring stack will use the same network to communicate with our application. If you haven't created it yet:

docker network create shared-network
Enter fullscreen mode Exit fullscreen mode

This network allows our monitoring services to communicate with both the application services and each other.

Grafana Configuration

Looking at the Grafana service, you'll notice several environment variables:

environment:
  - GF_SERVER_DOMAIN=your.domain
  - GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s/grafana
  - GF_SERVER_SERVE_FROM_SUB_PATH=true
Enter fullscreen mode Exit fullscreen mode

Replace your.domain with your actual domain name. These settings ensure Grafana works correctly under a subpath (/grafana) with your domain.

Service Configuration Files

Before starting the stack, you'll need configuration files for Prometheus, Loki, and Promtail:

  1. Create a prometheus.yml file in a new prometheus directory: 
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
      - targets: []
      scheme: http
      timeout: 10s
      api_version: v2

scrape_configs:
  - job_name: prometheus
    honor_timestamps: true
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /prometheus/metrics
    scheme: http
    static_configs:
      - targets: ['prometheus:9090']

  - job_name: cadvisor
    scrape_interval: 5s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ['cadvisor:8080']
Enter fullscreen mode Exit fullscreen mode
  1. Download Loki and Promtail configurations: 
# Get Loki config
wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml

# Get Promtail config
wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
Enter fullscreen mode Exit fullscreen mode

Nginx Proxy Manager Configuration

Add these locations to your existing proxy host configuration:

location /prometheus {
    proxy_pass http://prometheus:9090;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

location /grafana {
    proxy_pass http://grafana:3000;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}
Enter fullscreen mode Exit fullscreen mode

Nginx Proxy Manager Advanced Settings

Starting the Monitoring Stack

  1. Create required directories and set permissions: 
# Create directories
mkdir -p prometheus data

# Set Loki permissions (important!)
sudo chown -R 10001:10001 ./data
sudo chmod -R 755 ./data
Enter fullscreen mode Exit fullscreen mode
  1. Start the stack: 
docker compose -f monitoring-compose.yml up -d
Enter fullscreen mode Exit fullscreen mode
  1. Verify services are running: 
docker compose -f monitoring-compose.yml ps
Enter fullscreen mode Exit fullscreen mode

Accessing Your Monitoring Stack

Once everything is running, you can access:

  • Grafana: https://your.domain/grafana
  • Prometheus: https://your.domain/prometheus

Prometheus page

Grafana login page through subpath

Troubleshooting Tips

If you encounter issues:

  1. Check service logs: 
docker compose -f monitoring-compose.yml logs grafana
docker compose -f monitoring-compose.yml logs prometheus
Enter fullscreen mode Exit fullscreen mode
  1. Verify network connectivity: 
docker network inspect shared-network
Enter fullscreen mode Exit fullscreen mode

Setting Up Your Monitoring Dashboard

Now that our monitoring stack is running, let's configure our dashboards and data sources for effective monitoring.

Adding Data Sources to Grafana

A. Access Grafana at https://your.domain/grafana

  • Default credentials: admin/admin
  • You'll be prompted to change the password on first login

B. Configure Prometheus Data Source:

  1. Go to Settings (⚙️) > Data Sources
  2. Click "Add data source"
  3. Select "Prometheus"
  4. Configure:
    • Name: Prometheus
    • URL: http://prometheus:9090/prometheus #because of the sub-path requirements in our projects
    • Access: Server (default)
  5. Click "Save & Test"

Setting Up Loki for Log Aggregation

A. Add Loki Data Source:

  1. Go to Settings (⚙️) > Data Sources
  2. Click "Add data source"
  3. Select "Loki"
  4. Configure:
    • Name: Loki
    • URL: http://loki:3100
    • Access: Server (default)
  5. Click "Save & Test"

data source

cadvisor

loki logs

prometheus targets

References

  1. "My HNG Journey: Stage Two Containerization and Deployment of a Three-Tier Application Using Docker and Nginx Proxy Manager"

    A detailed guide on deploying a three-tier application with Docker and Nginx Proxy Manager, which influenced the reverse proxy configuration steps.

    Read more on dev.to

  2. Techdox Documentation

    Comprehensive documentation for setting up each monitoring services.

    Visit Docs Here

  3. Youtube videos

    Insightful YouTube videos with insight on deployment strategies and troubleshooting shared, particularly for Loki and Nginx.

Top comments (0)

Read next

atixag profile image

DevOps vs. Platform Engineering: Another Trend or the Next Big Thing?

ATIXAG -

vaish_98 profile image

Multicloud Cost Management Guide for FinOps Practitioners

Vaishnavi -

suin profile image

Sharing Secrets Between Kubernetes Clusters Using external-secrets PushSecret

suin -

ra_jeeves profile image

Building Brew Haven: A/B Testing My Coffee Shop Dreams with DevCycle

Rajeev R. Sharma -