DEV Community

shashankpai
shashankpai

Posted on

Unlocking Linux Host Performance Insights with Prometheus Monitoring (Part-1)

#prometheus #linux #monitoring #devops

Table Of Contents

Introduction:

Welcome to part 1 of our 2-part blog series on Linux host monitoring with Prometheus. This series is designed for DevOps professionals, monitoring enthusiasts, and Prometheus Certified Associate aspirants.

In this series, we'll explore the importance of monitoring Linux host metrics and how Prometheus can help. We'll cover topics such as exporting metrics through node exporter and how prometheus scrapes metrics.

Stay tuned for part 2, where we'll dive into practical implementation, configuration, and creating dashboards with grafana for monitoring Linux hosts with Prometheus.

Get ready to elevate your DevOps and monitoring skills as we unlock the power of Linux host monitoring with Prometheus. Let's begin this journey together.

Importance of monitoring Linux host metrics:

Monitoring Linux host metrics is crucial for optimizing performance, planning capacity, troubleshooting issues, ensuring security, and proactively maintaining systems. By tracking metrics like CPU, memory, disk, and network usage, you can identify problems, make data-driven decisions, and ensure the smooth operation of your Linux environment.

How Prometheus can help?

Prometheus is a powerful monitoring tool that simplifies the process of monitoring Linux host metrics. It collects data from various sources using lightweight exporters like Node Exporter and stores it in a time-series database. With PromQL, you can query and analyze the collected metrics, enabling you to retrieve specific Linux host metrics and gain insights into system performance.

What is Node Exporter and its role ?

Prometheus Server scraping metrics from Node Exporter service running on linux hosts
Prometheus Server scraping metrics from Node Exporter service running on Linux hosts

Linux doesn't provide native Prometheus metrics out of the box. However, Node Exporter comes to the rescue by bridging this gap. Node Exporter is an exporter specifically designed for Linux systems, serving as a reliable source for collecting Linux host metrics.

Node Exporter runs as a service on the Linux host, exposing a wide range of metrics related to system resources, network, disk, CPU, memory, and more. It periodically collects these metrics from the Linux kernel and other system sources, making them available for scraping by Prometheus.

By installing and configuring Node Exporter on Linux hosts, you can unlock a wealth of Linux-specific metrics that would otherwise be inaccessible to Prometheus. This includes detailed information about CPU utilization, memory usage, disk I/O, network traffic, and other crucial indicators of system performance.

Node Exporter acts as a bridge between the Linux operating system and Prometheus, ensuring that Linux host metrics can be easily monitored and analyzed using Prometheus' powerful features. It provides a standardized and reliable way to expose Linux-specific metrics, enabling seamless integration with the Prometheus ecosystem.

With Node Exporter, system administrators and DevOps teams can gain valuable insights into the performance and health of their Linux hosts, allowing them to proactively optimize resource allocation, troubleshoot issues, and ensure the smooth operation of their Linux-based infrastructure.

Setting up Prometheus server to monitor Linux hosts

Download and install Prometheus

1.Create the prometheus user 
sudo useradd -M -r -s /bin/false prometheus
Enter fullscreen mode Exit fullscreen mode
2.Create prometheus directories 
sudo mkdir /etc/prometheus /var/lib/prometheus
Enter fullscreen mode Exit fullscreen mode
3. Then download latest binary archive for Prometheus. 
mkdir -p /tmp/prometheus && cd /tmp/prometheus

curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -
Enter fullscreen mode Exit fullscreen mode
4. Untar the the downloaded binary 
tar xvf prometheus*.tar.gz
prometheus-2.45.0.linux-amd64/
prometheus-2.45.0.linux-amd64/LICENSE
prometheus-2.45.0.linux-amd64/prometheus.yml
prometheus-2.45.0.linux-amd64/console_libraries/
prometheus-2.45.0.linux-amd64/console_libraries/prom.lib
prometheus-2.45.0.linux-amd64/console_libraries/menu.lib
prometheus-2.45.0.linux-amd64/consoles/
prometheus-2.45.0.linux-amd64/consoles/node-overview.html
prometheus-2.45.0.linux-amd64/consoles/node-cpu.html
prometheus-2.45.0.linux-amd64/consoles/index.html.example
prometheus-2.45.0.linux-amd64/consoles/node.html
prometheus-2.45.0.linux-amd64/consoles/node-disk.html
prometheus-2.45.0.linux-amd64/consoles/prometheus-overview.html
prometheus-2.45.0.linux-amd64/consoles/prometheus.html
prometheus-2.45.0.linux-amd64/promtool
prometheus-2.45.0.linux-amd64/prometheus
prometheus-2.45.0.linux-amd64/NOTICE
Enter fullscreen mode Exit fullscreen mode
5. Move the binary files to /usr/local/bin/ directory and set their ownership to prometheus user. 
sudo mv prometheus promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/{prometheus,promtool}
Enter fullscreen mode Exit fullscreen mode
6. Move the files from the tmp archive to the appropriate locations, and set ownership on these files and directories to the prometheus user 
sudo cp -r prometheus-2.45.0.linux-amd64/{consoles,console_libraries} /etc/prometheus/

sudo cp prometheus-2.45.0.linux-amd64/prometheus.yml /etc/prometheus/prometheus.yml

sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
Enter fullscreen mode Exit fullscreen mode
7. Run Prometheus in the foreground to make sure everything is set up correctly so far: 
prometheus --config.file=/etc/prometheus/prometheus.yml
Enter fullscreen mode Exit fullscreen mode

Image description

Image description

8. Configure Prometheus as a systemd Service

Create a systemd unit file for Prometheus:

sudo vim /etc/systemd/system/prometheus.service
Enter fullscreen mode Exit fullscreen mode 
[Unit] Description=Prometheus Time Series Collection and Processing Server Wants=network-online.target After=network-online.target

[Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yml \ --storage.tsdb.path /var/lib/prometheus/ \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries 

[Install] WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode
9. Make sure systemd picks up the changes we made: 
sudo systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode
10. Start the Prometheus service: 
sudo systemctl start prometheus
Enter fullscreen mode Exit fullscreen mode
11. Enable the Prometheus service so it will automatically start at boot: 
sudo systemctl enable prometheus
Enter fullscreen mode Exit fullscreen mode
12. Verify the Prometheus service is healthy: 
sudo systemctl status prometheus
Enter fullscreen mode Exit fullscreen mode

We should see its state is active (running).

13. Make an HTTP request to Prometheus to verify it is able to respond:

curl localhost:9090

Image description

14. In a new browser tab, access Prometheus by navigating to http://:9090 (replacing with the IP listed on the lab page). We should then see the Prometheus expression browser.

Image description

Install and Configure Node Exporter on the Server

1. Create a user and group that will be used to run Node Exporter 
sudo useradd -M -r -s /bin/false node_exporter
Enter fullscreen mode Exit fullscreen mode
2. Get the node exporter binary 
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.0/node_exporter-1.6.0.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode
3. Extract the Node Exporter binary: 
tar -xvzf node_exporter-1.6.0.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode
4. Copy the Node Exporter binary to the appropriate location 
sudo cp node_exporter-1.6.0.linux-amd64/node_exporter /usr/local/bin/
Enter fullscreen mode Exit fullscreen mode
5. Set ownership on the Node Exporter binary 

sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
Enter fullscreen mode Exit fullscreen mode
6. Create a systemd unit file for Node Exporter: 
sudo vim /etc/systemd/system/node_exporter.service
Enter fullscreen mode Exit fullscreen mode
7.Define the Node Exporter service in the unit file 
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode
8. Make sure systemd picks up the changes we made: 
sudo systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode
9. Enable the node_exporter service so it will automatically start at boot: 
sudo systemctl enable node_exporter
Enter fullscreen mode Exit fullscreen mode
10. Test that your Node Exporter is working by making a request to it from localhost: 
curl localhost:9100/metrics
Enter fullscreen mode Exit fullscreen mode
11.Configure Prometheus to Scrape Metrics from the Acme Web Server

Edit the Prometheus config file:

sudo vim /etc/prometheus/prometheus.yml
Enter fullscreen mode Exit fullscreen mode

Locate the scrape_configs section and add the following beneath it (ensuring it's indented to align with the existing job_name section):

...

  - job_name: 'LimeDrop Web Server'
    static_configs:
    - targets: ['10.0.1.102:9100']

...
Enter fullscreen mode Exit fullscreen mode

Image description

11. Restart Prometheus to load the new config: 
sudo systemctl restart prometheus
Enter fullscreen mode Exit fullscreen mode
12. Navigate to the Prometheus expression browser in your web browser using the public IP address of your Prometheus server: 
<PROMETHEUS_SERVER_PUBLIC_IP>:9090
Enter fullscreen mode Exit fullscreen mode
13.In the expression field (the box at the top of the page), paste in the following query to verify you are able to get some metric data from the LimeDrop web server: 
node_filesystem_avail_bytes{job="Acme Web Server"}
Enter fullscreen mode Exit fullscreen mode

Image description

Image description

In the second part of our blog series on Linux host monitoring with Prometheus, we will delve into more advanced concepts of Node Exporter and explore how to create insightful dashboards and queries.

Top comments (0)

Read next

msnmongare profile image

How to Run a Cron Job Every 5 Minutes in Laravel 11

Sospeter Mong'are -

vivx_developer profile image

IPC Mechanisms in Linux

Vivek P -

clericcoder profile image

Mastering Trace Analysis with Span Links using openTelemetry and Signoz (A Practical Guide,Part 2)

Abdulsalaam Noibi -

clericcoder profile image

Mastering Trace Analysis with Span Links using OpenTelemetry and Signoz (A Practical Guide, Part 1)

Abdulsalaam Noibi -