DEV Community

Cover image for Prometheus & Grafana Installation with Node exporter,Alert Manager,SMTP service configuration for Monitoring
Kannan
Kannan

Posted on

Prometheus & Grafana Installation with Node exporter,Alert Manager,SMTP service configuration for Monitoring

#linux #devops #prometheus #grafana

Prometheus Installation

  • Do the apt Update
    apt update -y && apt upgrade -y

  • Download prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.47.2/prometheus-2.47.2.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode

tar -xvf prometheus-2.47.2.linux-amd64.tar.gz

  • Rename prometheus 2.47.2 to prometheus-files
    mv prometheus-2.47.2.linux-amd64 prometheus-files

  • Add prometheus user
    useradd --no-create-home --shell /bin/false prometheus

  • Create directories and addgroup
    mkdir /etc/prometheus
    mkdir /var/lib/prometheus
    groupadd prometheus

  • Change the ownership of the above created directories
    chown prometheus:prometheus /etc/prometheus
    chown prometheus:prometheus /var/lib/prometheus

  • Copy the prometheus and promtool to /etc/prometheus and provide ownership for the same
    cp -r prometheus-files/consoles /etc/prometheus
    cp -r prometheus-files/console_libraries /etc/prometheus
    chown -R prometheus:prometheus /etc/prometheus/consoles
    chown -R prometheus:prometheus /etc/prometheus/console_libraries

  • Create the prometheus.yml file and set the configuration
    vim /etc/prometheus/prometheus.yml

root@prometheus-1:/etc/prometheus# cat prometheus.yml 
global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'prometheus_server'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.122.181:9100']
Enter fullscreen mode Exit fullscreen mode

  • Change the ownership to prometheus user
    chown prometheus:prometheus /etc/prometheus/prometheus.yml

  • Create and setup a prometheus service file
    vim /etc/systemd/system/prometheus.service

root@prometheus-1:/etc/systemd/system# cat prometheus.service 
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
       --config.file /etc/prometheus/prometheus.yml \
       --storage.tsdb.path /var/lib/prometheus/ \
       --storage.tsdb.retention.time=7d \
       --storage.tsdb.retention.size=8GB \
       --web.console.templates=/etc/prometheus/consoles \
       --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode
  • Reload start and enable the prometheus 
systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus
systemctl status prometheus
Enter fullscreen mode Exit fullscreen mode
  • Access the prometheus on port 9090

Image description

Grafana Installation

  • Do apt update and download the Grafana GPG key apt update -y 
wget -q -O - https://packages.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > /dev/null
Enter fullscreen mode Exit fullscreen mode 
root@prometheus-1:/# apt update -y
Hit:1 http://in.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://in.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://in.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://in.archive.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
36 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@prometheus-1:/# wget -q -O - https://packages.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > /dev/null
Enter fullscreen mode Exit fullscreen mode
  • Install APT Grafana Repo 
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
Enter fullscreen mode Exit fullscreen mode 
root@prometheus-1:/# echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main
root@prometheus-1:/# apt update
Hit:1 http://in.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://in.archive.ubuntu.com/ubuntu jammy-updates InRelease
Get:3 https://packages.grafana.com/oss/deb stable InRelease [5,983 B]         
Hit:4 http://in.archive.ubuntu.com/ubuntu jammy-backports InRelease           
Hit:5 http://in.archive.ubuntu.com/ubuntu jammy-security InRelease
Get:6 https://packages.grafana.com/oss/deb stable/main amd64 Packages [163 kB]
Fetched 169 kB in 2s (70.4 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
36 packages can be upgraded. Run 'apt list --upgradable' to see them.
Enter fullscreen mode Exit fullscreen mode

  • Install Grafana
    apt install grafana -y

  • Start and enable the Grafana service

systemctl start grafana-server.service
systemctl enable grafana-server.service
systemctl status grafana-server.service
Enter fullscreen mode Exit fullscreen mode
  • Find Grafana version 
root@prometheus-1:/# grafana-server -v
Version 10.2.0 (commit: 895fbafb7a, branch: HEAD)
Enter fullscreen mode Exit fullscreen mode
  • Access the grafana dashboard on port 3000 login as user:admin & pass:admin and chage the credientials.

Image description

Image description

Node Exporter Installation

  • Download the node exporter 
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode

tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz

  • move the node exporter folder to /usr/local/bin
    mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/

  • create the group and add the user
    useradd -rs /bin/false node_exporter
    groupadd node_exporter

  • Create the Node exporter service file
    vim /etc/systemd/system/node_exporter.service

root@prometheus-1:/etc/systemd/system# cat node_exporter.service 
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode
  • reload, Enable & start the Node exporterservice 
systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter
systemctl status node_exporter
Enter fullscreen mode Exit fullscreen mode
  • Access the metrics on port 9100 Image description

Add Data source

Home >Dashboard>New>Importdashboard

Image description

Alert Manager Installation on Prometheus

  • apt update and download the Alert manager and copy the alert manager file to /usr/local/bin 
wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode

tar -xvf alertmanager-0.26.0.linux-amd64.tar.gz

cd alertmanager-0.26.0.linux-amd64/

cp -r . /usr/local/bin/alertmanager

  • Create the alert manager service file

vim /etc/systemd/system/alertmanager.service

root@prometheus-1:/etc/systemd/system# cat node_exporter.service 
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
root@prometheus-1:/etc/systemd/system# cat alertmanager.service 
[Unit]
Description=Prometheus Alert Manager Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/alertmanager/alertmanager \
    --config.file=/usr/local/bin/alertmanager/alertmanager.yml \
    --cluster.advertise-address="192.168.122.181:9093"

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode
  • Check the Alert manager with amtool 
/usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml
Enter fullscreen mode Exit fullscreen mode 
root@prometheus-1:~# /usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml
Checking '/usr/local/bin/alertmanager/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 1 receivers
 - 0 templates
Enter fullscreen mode Exit fullscreen mode
  • Reload,start & enable the Alert manager service 
systemctl daemon-reload
systemctl start alertmanager.service
systemctl enable alertmanager.service
Enter fullscreen mode Exit fullscreen mode
  • Access the Alert manager on port 9093

Image description

Add alert rules

  • adding alert managerconfiguration on prometheus.yml file which we created on /etc/prometheus/

add the below parameters before the scarpe_configs

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - localhost:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "/etc/prometheus/rules/*.yml"
Enter fullscreen mode Exit fullscreen mode

SMTP Configure for Alert manager

  • To proceed with these setup we need to enable 2-factor authentication on your "Gmail" account and create the "app password" for (prometheus)and copy the key to the below auth password paramater.

Image description

  • create and add the paramaters on alertmanager.yml
    vim /usr/local/bin/alertmanager/alertmanager.yml

  • Fill the "FROM","TO" email and add the auth key generated from the user gmail account.

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 24h
  receiver: 'email'
receivers:
- name: 'email'
  email_configs:
  - to: "recepient mail address"@gmail.com
    from: '"usermail address"@gmail.com'
    smarthost: smtp.gmail.com:587
    auth_username: '"usermail address"@gmail.com'
    auth_identity: '"usermail address".com'
    auth_password: '(Key we copied from the appsecurity tab)'
    send_resolved: true
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']
Enter fullscreen mode Exit fullscreen mode

Alert Manager Rule

  • Create the directories to add the rules.

mkdir /etc/prometheus/rules

vim /etc/prometheus/rules/alert-rules.yml

groups:
- name: alert-rules
  rules:
  - alert: ExporterDown
    expr: up == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      description: 'Metrics exporter service for {{ $labels.job }} running on {{ $labels.instance }} has been down for more than 5 minutes.'
      summary: 'Exporter down (instance {{ $labels.instance }})'

  - alert: HostOutOfDiskSpace
    expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 15 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host out of disk space (instance {{ $labels.instance }})
      description: "Disk is almost full (< 15% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: HostOutOfMemory
    expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 15
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host out of memory (instance {{ $labels.instance }})
      description: "Node memory is filling up (< 15% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: HostHighCpuLoad
    expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 85
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host high CPU load (instance {{ $labels.instance }})
      description: "CPU load is > 85%\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
Enter fullscreen mode Exit fullscreen mode

Image description

To check and find the status validation of alertmanager.yml and rules with amtool & promtool 

/usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml
Enter fullscreen mode Exit fullscreen mode 
promtool check rules /etc/prometheus/rules/alert-rules.yml
Enter fullscreen mode Exit fullscreen mode 
root@prometheus-1:~# /usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml
Checking '/usr/local/bin/alertmanager/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 1 receivers
 - 0 templates

root@prometheus-1:~# promtool check rules /etc/prometheus/rules/alert-rules.yml
Checking /etc/prometheus/rules/alert-rules.yml
  SUCCESS: 4 rules found
Enter fullscreen mode Exit fullscreen mode

Restart all the services like prometheus,node exporter,Alert manager,grafana

  • We have verified the disk space and added file.img to check the alertmanager and rules. 
root@prometheus-1:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              392M  1.3M  390M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   12G  7.3G  3.4G  69% /
tmpfs                              2.0G     0  2.0G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/vda2                          2.0G  251M  1.6G  14% /boot
tmpfs                              392M  4.0K  392M   1% /run/user/1000
root@prometheus-1:~# fallocate -l 2G file.img
root@prometheus-1:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              392M  1.3M  390M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   12G  9.3G  1.4G  88% /
tmpfs                              2.0G     0  2.0G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/vda2                          2.0G  251M  1.6G  14% /boot
tmpfs                              392M  4.0K  392M   1% /run/user/1000
root@prometheus-1:~# ls
Enter fullscreen mode Exit fullscreen mode

-Grafana Dashboard status.
Image description

Top comments (0)

Read next

pravesh_sudha_3c2b0c2b5e0 profile image

Optimise AWS Costs: Automate Unused EBS Snapshot Cleanup with Lambda

Pravesh Sudha -

signadot profile image

Why Staging Is a Bottleneck for Microservice Testing

Signadot -

msnmongare profile image

How to SSH into a Server Using IP Address, Username, and Password

Sospeter Mong'are -

giveitatry profile image

How to Install k3s with High Availability (HA)

giveitatry -