Monitoring ElasticSearch Cluster using Prometheus and Grafana

We are going to monitor our production ElasticSearch Cluster using elasticsearch exporter plugin and store those metrics on the Prometheus server and later visualize it using Grafana Dashboard. The exporter will scrape the metrics from the cluster and make it available on default port 9114(which can be changed) and Prometheus will scrape those metrics from the exporter and store it on its TSD.

ES Cluster >> es_exporter >> Prometheus >> Grafana

ElasticSearch Exporter:

ElasticSearch Exporter is a ElasticSearch stats exporter for Prometheus which is written in GO LANG.

NOTE: Exporter fetches metrics from an ElasticSearch cluster on every scrape, therefore having a too short scrape interval can impose load on ES master nodes, particularly if you run with --es.all and --es.indices.

Link for ElasticSearch Exporter github project.

I'm going to use pre-built binaries here.

wget https://github.com/justwatchcom/elasticsearch_exporter/releases/download/v1.1.0/elasticsearch_exporter-1.1.0.linux-amd64.tar.gz
tar xvf elasticsearch_exporter-1.1.0.linux-amd64.tar.gz
cd elasticsearch_exporter-1.1.0.linux-amd64
cp elasticsearch_exporter /usr/local/bin/es_exporter

If you need any further information regarding binary then use --help option.

elasticsearch_exporter --help

SYSTEMD SERVICE

Now create systemd service file for elasticsearch exporter in your ES server. Define your ES cluster username password if set.

vim /etc/systemd/system/es_exporter.service

[Unit]
Description=Prometheus ES_exporter
After=local-fs.target network-online.target network.target
Wants=local-fs.target network-online.target network.target

[Service]
User=root
Nice=10
ExecStart = /usr/local/bin/es_exporter --es.uri=http://elastic_user:XXXXXXXXXXX@localhost:9200 --es.all --es.indices --es.timeout 20s
ExecStop= /usr/bin/killall es_exporter

[Install]
WantedBy=default.target

systemctl start es_exporter.service
systemctl enable es_exporter.serivce

PROMETHEUS

config for prometheus.yml

vim /etc/prometheus/prometheus.yml
Add below lines.

  - job_name: elasticsearch
    scrape_interval: 60s
    scrape_timeout:  30s
    metrics_path: "/metrics"
    static_configs:
    - targets: ['es-01:9114' ]
      labels:
        service: elasticsearch
    relabel_configs:
    - source_labels: [__address__]
      regex: '(.*)\:9108'
      target_label:  'instance'
      replacement:   '$1'
    - source_labels: [__address__]
      regex:         '.*\.(.*)\.lan.*'
      target_label:  'environment'
      replacement:   '$1'

systemctl restart prometheus

GRAFANA DASHBOARD

Import https://grafana.com/grafana/dashboards/2322 Grafana Dashboard and start monitoring your ElasticSearch Cluster metrics. You can either download in JSON format and upload in your grafana server or copy ID and import directly.