DEV Community

Deepanshu Anand
Deepanshu Anand

Posted on

Alerts and Alert Manager in Prometheus

#monitoring #alertmanager #prometheu #gmail

In the last blog we walked through the process of setting up a Prometheus server with Grafana dashboard. In this blog we will be setting up the alertmanager in the prometheus server.

Structure of an alertmanager and the prometheus server

alertmanager1

Defining rules for prometheus server

  • In the last blog we created a alerts.rules config files, this file will contain all the rules for our prometheus server 
cd /opt/prometheus/
vim alerts.rules
Enter fullscreen mode Exit fullscreen mode 
groups:
  - name: Linux-disk-space
    rules:
     - alert: Linux-DiskSpaceAlert
       expr:  (node_filesystem_avail_bytes/node_filesystem_size_bytes)*100 < 20
       for: 5m
       annotations:
        summary: "Disk space is running low"
        description: 'The available disk space on {{ $labels.instance }} is less than 80percent'


  - name: Linux-memory-utilisation
    rules:
     - alert: Linux-MemoryUsage
       expr: ((node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes)*100 ) < 20
       for: 2m
       annotations:
        Summary: "High RAM Usage"
        description: 'Memory usage on {{ $labels.instance }} is greater than 80percent'

  - name: Linux-up
    rules:
     - alert: Linux-up-status
       expr: up{job='node-exporter'} < 1
       for: 2m
       annotations:
        Summary: "Linux machine/exporter down"
        description: '{{ $labels.instance }} metrics could not be fetched machine may be down'

  - name: Windows-memory-utilisation
    rules:
     - alert: Windows-MemoryUsage
       expr:  ((windows_os_physical_memory_free_bytes/windows_cs_physical_memory_bytes)*100 ) < 20
       for: 2m
       annotations:
        Summary: "High RAM Usage"
        description: 'RAM usage on {{ $labels.instance }} is higher than 80 percent'

  - name: Windows-diskspace-utilisation
    rules:
     - alert: Windows-diskspace-utilisation
       expr: (windows_logical_disk_free_bytes/windows_logical_disk_size_bytes)*100 < 20
       for: 2m
       annotations:
        Summary: "Disk Space running low"
        description: 'Diskspace utilisation on {{ $labels.instance }} is higher than 80 percent'

  - name: Windows-up
    rules:
     - alert: Windows-up-status
       expr: up{job='win-exporter'} < 1
       for: 2m
       annotations:
        Summary: "Windows machine/exporter down"
        description: '{{ $labels.instance }} metrics could not be fetched machine may be down'
Enter fullscreen mode Exit fullscreen mode

Setting the alertmanager

  • for this walkthrough we will be setting up gmail alerts using gmail smtp server

1. setting up gmail smtp server

To set up a gmail smtp server for prometheus we need a gmail account say sender@gmail.com on which 2-step verification is enabled

step-1

step-2

step-3

  • your app password is your secret-key which will be used in alertmanager.yml as auth_password step-4

2. config file for alertmanager 

cd /opt/prometheus/
touch alertmanager.yml
vim alertmanager.yml
Enter fullscreen mode Exit fullscreen mode 
global:
  resolve_timeout: 1m

route:
  receiver: 'gmail-alert-notification'
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 1m

# sender@gmail is the gmail account on which you have generated your app password
receivers:
- name: 'gmail-alert-notification'
  email_configs:
    - to: 'receiver@gmail.com'
      from: 'sender@gmail.com'
      smarthost: smtp.gmail.com:587
      auth_username: 'sender@gmail.com'
      auth_identity: 'sender@gmail.com'
      auth_password: 'secret-key'
Enter fullscreen mode Exit fullscreen mode

3. installing the alertmanager docker image

We will be installing the docker image of alertmanager on the same machine on which we installed the prometheus server

docker run -d -p 9093:9093 -v /opt/prometheus/alertmanager.yml:/etc/alertmanager/alertmanager.yml --restart unless-stopped --name alertmanager prom/alertmanager
Enter fullscreen mode Exit fullscreen mode
  • verify that alertmanager is running by visiting alertmanager-ip:9093 in your browser

Adding rules and alertmanager in the prometheus 

cd /opt/prometheus/
vim prometheus.yml
Enter fullscreen mode Exit fullscreen mode 
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['ubuntu_id:9100']

  - job_name: 'win-exporter'
    static_configs:
      - targets: ['windows_ip:9182']

rule_files:
  - "/etc/prometheus/alerts.rules"

alerting:
  alertmanagers:
    - static_configs:
      - targets:
        - "ubuntu_ip:9093"
# ubuntu_ip is the IP on which the alertmanager container is installed, use localcost:9093 if same as prometheus server
Enter fullscreen mode Exit fullscreen mode 
# if prometheus docker is already running run the following command
docker restart prometheus

# if not run the following
docker run -d -p 9090:9090 -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /opt/prometheus/alerts.rules:/etc/prometheus/alerts.rules --restart unless-stopped --name prometheus prom/prometheus
Enter fullscreen mode Exit fullscreen mode

Your prometheus and alertmanager is ready and can be verified by visiting prometheus_ip:9090 and alertmanager_ip:9093 on your browser

I hope this blog helped you understand how to setup prometheus with alertmanager. Stay connected for more such blogs

Top comments (0)

Read next

thepracticaldev profile image

Extending Winner Announcement Dates for DEV Challenges

dev.to staff -

uwadon1 profile image

Step-by-step guide to containerize your full-stack MERN application, using Docker Compose.

Uwadone Joshua -

sandeep_a67a84444cb5f3f6f profile image

Technical writing road map

Sandeep -

rabdeep_kaur_27fed89047ff profile image

How to tackle your first Hacktober Fest

Rabdeep Kaur -