In this article, we will look how we can configure HTTPS and Authentication on both Prometheus and Node Exporter
Prerequisites
In my previous article, we looked at how we can set up Prometheus and Node Exporter as systemd services on an Ubuntu instance.
But in the above setup, we were accessing the Prometheus expression browser and Node Exporter metrics endpoints via HTTP and there was no authentication enabled.
We are going to address these issues in this article.
HTTPS
Node Exporter
Create a new directory for storing the Node Exporter configuration file and change its ownership to "node_exporter" user
$ sudo mkdir -p /etc/node_exporter
$ sudo chown node_exporter:node_exporter /etc/node_exporter
Create a configuration file for Node Exporter and change its ownership to "node_exporter" user
$ sudo touch /etc/node_exporter/node_exporter.yml
$ sudo chown node_exporter:node_exporter /etc/node_exporter/node_exporter.yml
Generate a certificate and key using OpenSSL
$ sudo openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout prom.key -out prom.crt -subj "/C=US/ST=California/L=Oakland/O=MyOrg/CN=localhost" -addext "subjectAltName = DNS:localhost"
$ ls
prom.crt prom.key
Copy the certificate and key files to the Node Exporter configuration directory and change their ownership to "node_exporter" user
$ sudo cp prom.* /etc/node_exporter
$ sudo chown node_exporter:node_exporter prom.crt
$ sudo chown node_exporter:node_exporter prom.key
Add the tls_server_config details to the configuration file
$ sudo vi /etc/node_exporter/node_exporter.yml
tls_server_config:
cert_file: prom.crt
key_file: prom.key
Update the systemd unit file of node_exporter service to include the above configuration file
$ sudo vi /etc/node_exporter/node_exporter.yml
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
--web.config.file /etc/node_exporter/node_exporter.yml
[Install]
WantedBy=multi-user.target
Restart the node_exporter service and verify its status
$ sudo systemctl restart node_exporter
$ sudo systemctl status node_exporter
The metrics endpoint is now accessible via HTTPS
Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing down
Prometheus
Copy the certificate and key files to the Prometheus configuration directory and change its ownership to "prometheus" user
$ sudo cp prom.* /etc/prometheus
$ sudo chown prometheus:prometheus /etc/prometheus/prom.crt
$ sudo chown prometheus:prometheus /etc/prometheus/prom.key
Update the Prometheus configuration file to include scheme and tls_config for the "node_exporter" job
$ sudo vi /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s
scrape_timeout: 10s
scrape_configs:
- job_name: "node_exporter"
scheme: https
tls_config:
ca_file: prom.crt
insecure_skip_verify: true
static_configs:
- targets: ["172.31.81.113:9100"]
Validate the configuration file using the promtool
$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
Restart the prometheus service to take effect the new configuration changes
$ sudo systemctl restart prometheus
$ sudo systemctl status prometheus
Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing up
Now we have enabled secure communication between the Prometheus server and Node Exporter but still our Prometheus expression browser is using an HTTP connection
Create a new configuration file for configuring HTTPS connection and change its ownership to "prometheus" user
$ sudo touch /etc/prometheus/webconfig.yml
$ sudo chown prometheus:prometheus /etc/prometheus/webconfig.yml
Add the tls_server_config details to the newly created configuration file
$ sudo vi /etc/prometheus/webconfig.yml
tls_server_config:
cert_file: prom.crt
key_file: prom.key
Update the systemd unit file of prometheus service to include the above configuration file
$ sudo vi /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus \
--web.console.templates /etc/prometheus/consoles \
--web.console.libraries /etc/prometheus/console_libraries \
--web.config.file /etc/prometheus/webconfig.yml
[Install]
WantedBy=multi-user.target
Now we can access the Prometheus expression browser using HTTPS
Authentication
Install the apache2 utils package to generate a password
$ sudo apt update
$ sudo apt install apache2-utils
Generate the password using the htpasswd tool
$ htpasswd -nBC 16 admin | tr -d ':\n'
New password:
Re-type new password:
admin$2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O
Node Exporter
Update the Node Exporter configuration file to include basic authentication
$ sudo vi /etc/node_exporter/node_exporter.yml
tls_server_config:
cert_file: prom.crt
key_file: prom.key
basic_auth_users:
admin: $2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O
Restart the node_exporter service and verify its status
$ sudo systemctl restart node_exporter
$ sudo systemctl status node_exporter
Now access the Node Exporter metrics endpoint and it will show a login prompt
Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing down
Prometheus
Update the Prometheus configuration file to include basic authentication for the "node_exporter" job
$ sudo vi /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s
scrape_timeout: 10s
scrape_configs:
- job_name: "node_exporter"
scheme: https
tls_config:
ca_file: prom.crt
insecure_skip_verify: true
basic_auth:
username: admin
password: Password!
static_configs:
- targets: ["172.31.81.113:9100"]
Validate the configuration file using the promtool
$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
Restart the prometheus service to effect new configuration changes
$ sudo systemctl restart prometheus
$ sudo systemctl status prometheus
Open the Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing up
Now we have enabled basic authentication between the Prometheus server and Node Exporter but we need to enable authentication for the Prometheus server
Update the below configuration file to include basic authentication on the Prometheus server
$ sudo vi /etc/prometheus/webconfig.yml
tls_server_config:
cert_file: prom.crt
key_file: prom.key
basic_auth_users:
admin: $2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O
Restart the prometheus service for the new changes and check its status
$ sudo systemctl restart prometheus
$ sudo systemctl status prometheus
Now access the Prometheus expression browser and it will show a login prompt
That's all for now
Reference
https://prometheus.io/docs/prometheus/latest/configuration/https/
https://kodekloud.com/courses/prometheus-certified-associate-pca/
Top comments (0)