Kannan

Posted on Nov 2, 2023

Prometheus & Grafana Installation with Node exporter,Alert Manager,SMTP service configuration for Monitoring

#linux #devops #prometheus #grafana

Prometheus Installation

Do the apt Update
apt update -y && apt upgrade -y
Download prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.47.2/prometheus-2.47.2.linux-amd64.tar.gz

tar -xvf prometheus-2.47.2.linux-amd64.tar.gz

Rename prometheus 2.47.2 to prometheus-files
mv prometheus-2.47.2.linux-amd64 prometheus-files
Add prometheus user
useradd --no-create-home --shell /bin/false prometheus
Create directories and addgroup
mkdir /etc/prometheus
mkdir /var/lib/prometheus
groupadd prometheus
Change the ownership of the above created directories
chown prometheus:prometheus /etc/prometheus
chown prometheus:prometheus /var/lib/prometheus
Copy the prometheus and promtool to /etc/prometheus and provide ownership for the same
cp -r prometheus-files/consoles /etc/prometheus
cp -r prometheus-files/console_libraries /etc/prometheus
chown -R prometheus:prometheus /etc/prometheus/consoles
chown -R prometheus:prometheus /etc/prometheus/console_libraries
Create the prometheus.yml file and set the configuration
vim /etc/prometheus/prometheus.yml

root@prometheus-1:/etc/prometheus# cat prometheus.yml 
global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'prometheus_server'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.122.181:9100']

Change the ownership to prometheus user
chown prometheus:prometheus /etc/prometheus/prometheus.yml
Create and setup a prometheus service file
vim /etc/systemd/system/prometheus.service

root@prometheus-1:/etc/systemd/system# cat prometheus.service 
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
       --config.file /etc/prometheus/prometheus.yml \
       --storage.tsdb.path /var/lib/prometheus/ \
       --storage.tsdb.retention.time=7d \
       --storage.tsdb.retention.size=8GB \
       --web.console.templates=/etc/prometheus/consoles \
       --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Reload start and enable the prometheus

systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus
systemctl status prometheus

Access the prometheus on port 9090

Grafana Installation

Do apt update and download the Grafana GPG key apt update -y

wget -q -O - https://packages.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > /dev/null

root@prometheus-1:/# apt update -y
Hit:1 http://in.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://in.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://in.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://in.archive.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
36 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@prometheus-1:/# wget -q -O - https://packages.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > /dev/null

Install APT Grafana Repo

echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

root@prometheus-1:/# echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main
root@prometheus-1:/# apt update
Hit:1 http://in.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://in.archive.ubuntu.com/ubuntu jammy-updates InRelease
Get:3 https://packages.grafana.com/oss/deb stable InRelease [5,983 B]         
Hit:4 http://in.archive.ubuntu.com/ubuntu jammy-backports InRelease           
Hit:5 http://in.archive.ubuntu.com/ubuntu jammy-security InRelease
Get:6 https://packages.grafana.com/oss/deb stable/main amd64 Packages [163 kB]
Fetched 169 kB in 2s (70.4 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
36 packages can be upgraded. Run 'apt list --upgradable' to see them.

Install Grafana
apt install grafana -y
Start and enable the Grafana service

systemctl start grafana-server.service
systemctl enable grafana-server.service
systemctl status grafana-server.service

Find Grafana version

root@prometheus-1:/# grafana-server -v
Version 10.2.0 (commit: 895fbafb7a, branch: HEAD)

Access the grafana dashboard on port 3000 login as user:admin & pass:admin and chage the credientials.

Node Exporter Installation

Download the node exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz

tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz

move the node exporter folder to /usr/local/bin
mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
create the group and add the user
useradd -rs /bin/false node_exporter
groupadd node_exporter
Create the Node exporter service file
vim /etc/systemd/system/node_exporter.service

root@prometheus-1:/etc/systemd/system# cat node_exporter.service 
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

reload, Enable & start the Node exporterservice

systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter
systemctl status node_exporter

Access the metrics on port 9100

Add Data source

You can get the data source from below link
[(https://grafana.com/grafana/dashboards/?plcmt=footer)]
You can add the ID or upload the JSON package on the grafana dashboard.

Home >Dashboard>New>Importdashboard

Alert Manager Installation on Prometheus

apt update and download the Alert manager and copy the alert manager file to /usr/local/bin

wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz

tar -xvf alertmanager-0.26.0.linux-amd64.tar.gz

cd alertmanager-0.26.0.linux-amd64/

cp -r . /usr/local/bin/alertmanager

Create the alert manager service file

vim /etc/systemd/system/alertmanager.service

root@prometheus-1:/etc/systemd/system# cat node_exporter.service 
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
root@prometheus-1:/etc/systemd/system# cat alertmanager.service 
[Unit]
Description=Prometheus Alert Manager Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/alertmanager/alertmanager \
    --config.file=/usr/local/bin/alertmanager/alertmanager.yml \
    --cluster.advertise-address="192.168.122.181:9093"

[Install]
WantedBy=multi-user.target

Check the Alert manager with amtool

/usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml

root@prometheus-1:~# /usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml
Checking '/usr/local/bin/alertmanager/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 1 receivers
 - 0 templates

Reload,start & enable the Alert manager service

systemctl daemon-reload
systemctl start alertmanager.service
systemctl enable alertmanager.service

Access the Alert manager on port 9093

Add alert rules

adding alert managerconfiguration on prometheus.yml file which we created on /etc/prometheus/

add the below parameters before the scarpe_configs

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - localhost:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "/etc/prometheus/rules/*.yml"

SMTP Configure for Alert manager

To proceed with these setup we need to enable 2-factor authentication on your "Gmail" account and create the "app password" for (prometheus)and copy the key to the below auth password paramater.

create and add the paramaters on alertmanager.yml
vim /usr/local/bin/alertmanager/alertmanager.yml
Fill the "FROM","TO" email and add the auth key generated from the user gmail account.

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 24h
  receiver: 'email'
receivers:
- name: 'email'
  email_configs:
  - to: "recepient mail address"@gmail.com
    from: '"usermail address"@gmail.com'
    smarthost: smtp.gmail.com:587
    auth_username: '"usermail address"@gmail.com'
    auth_identity: '"usermail address".com'
    auth_password: '(Key we copied from the appsecurity tab)'
    send_resolved: true
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

Alert Manager Rule

Create the directories to add the rules.

mkdir /etc/prometheus/rules

vim /etc/prometheus/rules/alert-rules.yml

groups:
- name: alert-rules
  rules:
  - alert: ExporterDown
    expr: up == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      description: 'Metrics exporter service for {{ $labels.job }} running on {{ $labels.instance }} has been down for more than 5 minutes.'
      summary: 'Exporter down (instance {{ $labels.instance }})'

  - alert: HostOutOfDiskSpace
    expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 15 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host out of disk space (instance {{ $labels.instance }})
      description: "Disk is almost full (< 15% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: HostOutOfMemory
    expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 15
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host out of memory (instance {{ $labels.instance }})
      description: "Node memory is filling up (< 15% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: HostHighCpuLoad
    expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 85
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host high CPU load (instance {{ $labels.instance }})
      description: "CPU load is > 85%\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

To check and find the status validation of alertmanager.yml and rules with amtool & promtool

/usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml

promtool check rules /etc/prometheus/rules/alert-rules.yml

root@prometheus-1:~# /usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml
Checking '/usr/local/bin/alertmanager/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 1 receivers
 - 0 templates

root@prometheus-1:~# promtool check rules /etc/prometheus/rules/alert-rules.yml
Checking /etc/prometheus/rules/alert-rules.yml
  SUCCESS: 4 rules found

Restart all the services like prometheus,node exporter,Alert manager,grafana

We have verified the disk space and added file.img to check the alertmanager and rules.

root@prometheus-1:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              392M  1.3M  390M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   12G  7.3G  3.4G  69% /
tmpfs                              2.0G     0  2.0G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/vda2                          2.0G  251M  1.6G  14% /boot
tmpfs                              392M  4.0K  392M   1% /run/user/1000
root@prometheus-1:~# fallocate -l 2G file.img
root@prometheus-1:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              392M  1.3M  390M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   12G  9.3G  1.4G  88% /
tmpfs                              2.0G     0  2.0G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/vda2                          2.0G  251M  1.6G  14% /boot
tmpfs                              392M  4.0K  392M   1% /run/user/1000
root@prometheus-1:~# ls

-Grafana Dashboard status.

DEV Community

Prometheus & Grafana Installation with Node exporter,Alert Manager,SMTP service configuration for Monitoring

Prometheus Installation

Grafana Installation

Node Exporter Installation

Add Data source

Alert Manager Installation on Prometheus

Add alert rules

Alert Manager Rule

To check and find the status validation of alertmanager.yml and rules with amtool & promtool

Top comments (0)

Read next

How to install Wave AI Terminal on Linux

Boost Software Development Success: Integrating Cross-Disciplinary Skill Sets for Better Results

Playing with Rust: Building a Safer rm and Having Fun Along the Way

Buttercup Password Manager for Startups 🛡️