Day 3 of My 90 Days of DevOps Projects: Monitoring with Prometheus and Grafana

Hi guys,

I am a bit late today to Day 3 of my 90 days of DevOps project building. I am one of the participants in the ongoing Zero to Merge open source projects organized by CNCF, and it's my first time juggling that with other life issues. It's quite interesting, but we move! Now, I am here, and today my focus was on Prometheus for monitoring and Grafana for visualization and alerting.

What Do Prometheus and Grafana Do?

Prometheus and Grafana are like having a health monitor for your applications. Prometheus collects data about how well your applications are running, such as response times and error rates. Grafana then takes this data and displays it in beautiful, easy-to-read graphs and charts. This helps you see at a glance if your applications are running smoothly or if something needs attention.

What You'll Gain from This Task

By the end of this task, you’ll learn how to:

• Set up a Node.js application that Prometheus can monitor.
• Use Docker to run Prometheus and Grafana.
• Create a Grafana dashboard to visualize metrics collected by Prometheus.
• Understand the basics of monitoring and why it's crucial in DevOps.
• Use GitHub Actions for CI/CD to build and deploy Docker images.

What You Need

Before we dive in, ensure you have the following:

• Docker installed on your computer.
• A GitHub account for repository management.
• Basic knowledge of Node.js and Docker.
• Access to Docker Hub.

Step-by-Step Guide

Step 1: Set Up a Sample Node.js Application

1. Create a Simple Node.js Application:

• Initialize a new Node.js project and install the necessary dependencies.
• Create a server that exposes some metrics for Prometheus.

   const express = require('express');
   const app = express();
   const port = 4000;

   app.get('/', (req, res) => res.send('Hello World!'));

   app.listen(port, () => console.log(`Example app listening at http://localhost:${port}`));

2. Add Prometheus Metrics:

• Use the prom-client package to collect and expose metrics.

   const client = require('prom-client');
   const collectDefaultMetrics = client.collectDefaultMetrics;

   collectDefaultMetrics();

   app.get('/metrics', (req, res) => {
     res.set('Content-Type', client.register.contentType);
     res.end(client.register.metrics());
   });

3. Dockerize Your Application:

• Create a Dockerfile to containerize your Node.js application.

   FROM node:14
   WORKDIR /app
   COPY package*.json ./
   RUN npm install
   COPY . .
   EXPOSE 4000
   CMD ["node", "index.js"]

4. Build and Push Docker Image:

• Use Docker commands to build and push your Docker image to Docker Hub.

   docker build -t your_dockerhub_username/my-monitoring-app:latest .
   docker push your_dockerhub_username/my-monitoring-app:latest

Step 2: Set Up GitHub Secrets for Docker Login

1. Create GitHub Secrets:
   • Go to your GitHub repository, click on Settings, then Secrets and Actions.
   • Add the following secrets:
     • DOCKER_USERNAME: Your Docker Hub username.
     • DOCKER_PASSWORD: Your Docker Hub password.

Step 3: Configure GitHub Actions for CI/CD

1. Create a GitHub Actions Workflow:
   • Create a file .github/workflows/docker-deploy.yml with the following content:

   name: Docker Build & Deploy

   on:
     push:
       branches:
         - main

   jobs:
     build-and-deploy:
       runs-on: ubuntu-latest
       steps:
         - name: Checkout Repository
           uses: actions/checkout@v2

         - name: Login to Docker Hub
           uses: docker/login-action@v2
           with:
             username: ${{ secrets.DOCKER_USERNAME }}
             password: ${{ secrets.DOCKER_PASSWORD }}

         - name: Build and Push Docker Image
           run: |
             docker build -t ${{ secrets.DOCKER_USERNAME }}/my-monitoring-app:latest .
             docker push ${{ secrets.DOCKER_USERNAME }}/my-monitoring-app:latest

         - name: Deploy Docker Container
           run: |
             docker-compose up -d

Step 4: Set Up Prometheus and Grafana with Docker Compose

1. Prometheus Configuration:

• Create a prometheus.yml file to define how Prometheus will scrape metrics from your Node.js application.

   global:
     scrape_interval: 15s

   scrape_configs:
     - job_name: 'node-app'
       static_configs:
         - targets: ['host.docker.internal:4000']

2. Create a docker-compose.yml File:

• Define services for Prometheus and Grafana.

   version: '3'

   services:
     prometheus:
       image: prom/prometheus
       volumes:
         - ./prometheus.yml:/etc/prometheus/prometheus.yml
       ports:
         - "9090:9090"

     grafana:
       image: grafana/grafana
       ports:
         - "3000:3000"

3. Run Docker Compose:

• Use Docker Compose to start Prometheus and Grafana.

   docker-compose up -d

Step 5: Create a Dashboard in Grafana

1. Access Grafana:

   • Open your browser and navigate to http://localhost:3000.
   • Log in with the default credentials (Username: admin, Password: admin).

2. Add Prometheus as a Data Source:

   • Go to Configuration > Data Sources > Add data source > Prometheus.
   • Set the URL to http://prometheus:9090 and click Save & Test.

3. Create a New Dashboard:

   • Click on the + (plus sign) in the left sidebar and select Dashboard.
   • Click Add new panel to create a new panel.

4. Configure the Panel:

   • Set the Panel title to something like "HTTP Request Duration".
   • In the Query section, select your Prometheus data source and enter a query like http_request_duration_ms_bucket.
   • Customize the visualization settings to your preference and click Apply.

5. Save the Dashboard:

   • After configuring the panel, click Save dashboard at the top right, give it a name, and save it.

Challenges and Solutions

Challenge 1: Docker Daemon Not Running

• Problem: Received an error indicating that the Docker daemon is not running.
• Solution: Ensure Docker Desktop is running and restart the Docker service.

Challenge 2: Port Already Allocated

• Problem: Encountered an error that the port 3000 is already allocated.
• Solution: Stop any service using the port 3000 and changed the Grafana port in the docker-compose.yml file to 3100.

Challenge 3: Grafana Panel Configuration Confusion

• Problem: Confusion about how to add a panel in Grafana.
• Solution: Detailed step-by-step instructions were provided to create and configure a panel in Grafana.

Challenge 4: Setting Up GitHub Secrets for Docker Login

• Problem: Difficulty in setting up GitHub secrets for Docker login.
• Solution: Provided a detailed guide on adding Docker Hub credentials as GitHub secrets.

How to Add This Project to GitHub

1. Initialize a Git Repository:
   • Navigate to your project directory and initialize a git repository.

   git init

1. Commit Your Changes:
   • Add and commit your project files.

   git add .
   git commit -m "Initial commit"

1. Push to GitHub:
   • Create a new repository on GitHub and push your local repository.

   git remote add origin https://github.com/your_username/your_repository_name.git
   git push -u origin main

Conclusion

It’s not easy, but I will push through till the end. Thanks for stopping by, and see you on Day 4 in a bit!

Thanks to my amazing community that goes over and beyond @shecodeAfrica and my amazing mentor @cornelia for her great support.

In this project, I set up a monitoring system using Prometheus and Grafana for a Node.js application. This task involved Dockerizing the application, running Prometheus and Grafana using Docker Compose, creating a Grafana dashboard to visualize metrics, and configuring GitHub Actions for CI/CD. Through these steps, I learned the importance of monitoring in DevOps and how to set up and configure essential monitoring tools.

By following this guide, you should now have a working monitoring setup for your own applications, and you can build on this foundation to add more sophisticated monitoring and alerting as needed. Happy monitoring!