Building a Docker-based Monitoring & Alerting System with Prometheus and Grafana (part-1)

Keeping your systems healthy is vital, and nothing beats being the first to know when something goes wrong. That’s where monitoring tools like Prometheus and Grafana come into play. In this article, we’ll walk through how to set up a monitoring and alerting system using Docker, to track both system metrics and website health.

By the end of this, you’ll have a fully operational monitoring stack using Prometheus, Grafana, Node Exporter (for system monitoring), and Blackbox Exporter (for website monitoring).

So, let's dive in!

Overview of the Monitoring Stack Architecture

Here’s a quick glance at the tools we’ll be using:

• Prometheus: It will handle data scraping and storage, collecting metrics from various exporters.
• Grafana: This will help you visualize the collected data in fancy dashboards.
• Node Exporter: This exporter collects system metrics, such as CPU usage, memory utilization, and disk I/O, from the machine on which it is installed. It’s perfect for getting detailed visibility into your server’s health.
• Blackbox Exporter: This exporter is used to monitor the availability and responsiveness of external services and websites. It can probe endpoints using HTTP, HTTPS, DNS, TCP, and ICMP to ensure your services are reachable.

Prerequisites

Before you get started, ensure you have the following installed:

• Docker: The platform we’ll use to run Prometheus, Grafana, and exporters in containers.
• Docker Compose: To manage our multi-container setup.

Helpful Resources:

• If you’re new to Docker, check out the Docker Get Started Guide.
• Learn how to install and use Docker Compose here.

Once you have these, we’re ready to go.

Step 1: Set Up Prometheus

• Create a working directory to store your configuration files.

# Here’s how the folder structure for your monitoring setup will look:

prometheus-grafana-monitoring-alerting/
├── docker-compose.yml
└── prometheus/
    └── prometheus.yml

• Create the Prometheus configuration file prometheus/prometheus.yml to define how it will scrape data:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

• Add Prometheus to Docker Compose docker-compose.yml:

version: '3.8'

volumes:
  prometheus-data: {}
  grafana-data: {}
  alertmanager-data: {}

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - '9090:9090'
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--web.enable-lifecycle'
      - '--config.file=/etc/prometheus/prometheus.yml'

• Run the docker-compose up -d command to start Prometheus.

Now, Prometheus will be accessible at http://localhost:9090 (Replace 'localhost' with the IP address). You'll add more targets to this setup in the following steps.

Step 2: Set Up Grafana

Grafana helps visualize the data scraped by Prometheus. Let’s set it up next.

• Update Docker Compose to include Grafana in docker-compose.yml:

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - '3000:3000'
    restart: unless-stopped
    volumes:
      - grafana-data:/var/lib/grafana

• Start Grafana by running docker-compose up -d.

Access Grafana via http://localhost:3000 (Replace 'localhost' with the IP address) with default credentials (admin/admin).

Adding Prometheus as a Data Source in Grafana:

• After logging into Grafana at http://localhost:3000 (or the IP address of your Grafana server), go to the Configuration in the left-hand menu.
• From the dropdown menu in Connections, select Data Source.
• Select Add data source.
• From the list, choose Prometheus.
• In the URL field, enter http://:9090 (use the Prometheus Ip address as defined in docker-compose.yml).
• Click Save & Test to ensure that Grafana can successfully connect to Prometheus.

Step 3: Add Node Exporter for System Monitoring

Now, let’s monitor your system’s CPU, memory, and disk usage using Node Exporter.

• Add Node Exporter to your Docker Compose file docker-compose.yml:

  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.rootfs=/host'
    network_mode: host
    pid: host
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'

• Update Prometheus configuration prometheus/prometheus.yml to scrape data from Node Exporter:

- job_name: 'node'
  static_configs:
    - targets: ['nodeexporter:9100']

• Run the updated Docker Compose with docker-compose up -d.

Your system metrics will now start flowing into Prometheus. Don’t forget to set up dashboards in Grafana to visualize these metrics.

For detailed instructions on setting up a Grafana dashboard for Node Exporter, refer to this guide: How to Add a Node Exporter Dashboard in Grafana.

Step 4: Set Up Blackbox Exporter for Website Monitoring

Next, we’ll monitor external websites or services with Blackbox Exporter.

• Add Blackbox Exporter to Docker Compose docker-compose.yml:

  blackbox-exporter:
    image: quay.io/prometheus/blackbox-exporter:latest
    container_name: blackbox_exporter
    ports:
      - '9115:9115'
    restart: unless-stopped
    volumes:
      - ./blackbox:/config
    command:
      - '--config.file=/config/blackbox.yml'

• Update Prometheus configuration to add website monitoring prometheus/prometheus.yml:

- job_name: 'blackbox'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - http://yourwebsiteurl-1.com
      - http://yourwebsiteurl-2.com
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - target_label: __address__
      replacement: <ip addr>:9115

(Note on IP Addresses: In this section, replace with the IP address of the machine running the Blackbox Exporter. If you’re running everything locally, you can replace with localhost. If this is a remote server, use the external IP address. Ensure that your firewall rules allow traffic on port 9115.)

• Restart Docker Compose docker-compose up -d.

For detailed steps on setting up a Grafana dashboard for Blackbox Exporter, check out this guide: How to Add a Blackbox Exporter Dashboard in Grafana.

Now, you’re monitoring both system metrics and website health!

Complete docker-compose.yml File,

version: '3.8'

volumes:
  prometheus-data: {}
  grafana-data: {}
  alertmanager-data: {}

services:
  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.rootfs=/host'
    network_mode: host
    pid: host
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - '9090:9090'
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--web.enable-lifecycle'
      - '--config.file=/etc/prometheus/prometheus.yml'

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - '3000:3000'
    restart: unless-stopped
    volumes:
      - grafana-data:/var/lib/grafana

  blackbox-exporter:
    image: quay.io/prometheus/blackbox-exporter:latest
    container_name: blackbox_exporter
    ports:
      - '9115:9115'
    restart: unless-stopped
    volumes:
      - ./blackbox:/config
    command:
      - '--config.file=/config/blackbox.yml'

Complete prometheus.yml File

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: "server-1"
    static_configs:
      - targets: ["<IP Addr>:9200"]

  # BlackBox Configuration
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  
    static_configs:
      - targets:
        - http://yourwebsiteurl-1.com
        - https://yourwebsiteurl-2.com

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: <ip addr>:9115

Bonus: Automated Monitoring Setup Script

For those who prefer an automated solution, I’ve created a Bash script available on GitHub that will set up the entire monitoring environment for you in just a few commands. This script is compatible with most Linux distributions where Docker and Docker Compose are already installed.

Simply run the following commands to download and execute the script:

git clone https://github.com/yagyandatta/infra-setup-scripts.git
cd infra-setup-scripts/monitoring_alerting
chmod +x setup.sh
./setup.sh

The script will:

• Pull the necessary Docker images for Prometheus, Grafana, Node Exporter, and Blackbox Exporter.
• Create the appropriate directory structure for storing configurations.
• Generate the basic configuration files for Prometheus and Grafana.
• Automatically bring up the containers using Docker Compose.

The script takes care of everything. The only prerequisite is having Docker and Docker Compose installed.

Conclusion

Congratulations! You now have a fully functioning monitoring setup using Prometheus and Grafana, with both system and website monitoring capabilities.

What’s Next?
Now that you have a fully operational monitoring system, you can further enhance it by:

• Customizing alerts to receive notifications when certain conditions are met (e.g., CPU usage exceeds a threshold).
• Building custom Grafana dashboards tailored to your needs.
• Exploring more Prometheus exporters to monitor additional systems and services.

Have any feedback or questions? Drop them in the comments below, and stay tuned for Part 2, where we’ll dive into creating customized alerts and advanced Grafana dashboards.

Stay tuned!