Use Grafana to plot Express.js apps Metrics

In this blog get-prometheus-metrics-from-a-express-js-app I exposed the metrics from an Express.js app to Prometheus. Now I am going to used these metrics in Grafana.

Install Grafana

Create a new project

oc new-project grafana

We can use the OperatorHub in Red Hat Openshift to install the Grafana-operator. I have logged in as kubeadmin user,and navigate to the OperatorHub and search for Grafana

Select the Grafana tile and continue on the install screen select install button

In the next screen select the grafana namespace and click install again

The operator should complete installing once finished go to the view operator button

We need to create a CR(custom resource) for Grafana to create a Grafana instance in the Grafana tile click on the create instance link

This will bring you to a form to populate the Grafana CR I just add a name and click create button

Thats it the Grafana instance should start, there is also a route created in the Grafana namespace.

Connecting to prometheus

Open the Grafana route in the browser

oc project grafana
oc get routes
NAME            HOST/PORT                                PATH   SERVICES          PORT   TERMINATION   WILDCARD
grafana-route   grafana-route-grafana.apps-crc.testing          grafana-service   3000   edge          None
# the HOST/PORT is the route so http://grafana-route-grafana.apps-crc.testing should open the grafana console

We will need to login to Grafana to be able to do anything. You can get the credentials for grafana in the Grafana CR

oc get grafana grafana -o yaml | grep admin
            f:admin_password: {}
            f:admin_user: {}
      admin_password: secret
      admin_user: root
# You can edit the secret in the Grafana CR also to change it from the default. 

Select the Sign In on the bottom of the screen

And used the admin credentials in the login screen

We can now connect Grafana to Prometheus by adding a data source. Go to the now available gear icon, and select Data Source and select Prometheus as show below

In the Data Source form in the HTML section add url for the Prometheus service. The service url is in the following format.

# service-name.service-namespace.svc:port
http://prometheus.default.svc:9090

You can now see prometheus metrics in grafana.

Add some useful metrics to Grafana

You can play around with the Grafana UI to get familiar with it creating dashboards and panels or read the Grafana docs. Mainly its about adding the Prometheus expressions and pointing at the right data source.

http_request_duration_seconds_bucket
One of the metrics we get from the Express.js app is http_request_duration_seconds_bucket. When we use this metric Grafana prompts us to use Histogram_quantile with this metric.

# Prometheus Expression aggregates the 95th percentile
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

As you can see I'm not seeing anything on the graph. This is because I am not generating any traffic. I wrote a small script to hit the endpoints

while true; 
do
    curl http://example-app-default.apps-crc.testing/
    curl http://example-app-default.apps-crc.testing/hello
    curl -X POST -H "Content-Type: application/json" -d '{"name": "test", "email": "test@example.com"}' http://example-app-default.apps-crc.testing/bye
done

After a few minutes I am seeing the metric show up

Up
I use this metric to determine if the container pods are up and running. As there should be 3 pods we sum the up's and divide by 3 to get a single metric , and add it to Grafana as a gauge panel

# Prometheus expression 
sum(up)/3

Average request duration Express-prometheus
The Average request duration can be got by the following expression see prometheus docs for more info.

# Prometheus expression
rate(http_request_duration_seconds_sum[5m])/ rate(http_request_duration_seconds_count[5m])

Memory Metrics
There are a lot of memory metrics exposed by the Express.js app.

You can use any of these metrics for a panel.
In my example will use the Memory Heap Used vs Heap Total.

# Prometheus expressions
nodejs_heap_size_used_bytes
nodejs_heap_size_total_bytes

Adding the two metric to the panel

CPU Metrics
With CPU again there are a few metrics exposed from the Express.js app. Again when we add the CPU metric Grafana prompts us to use these metrics with rate

## Prometheus expressions
rate(process_cpu_seconds_total[5m])
rate(process_cpu_system_seconds_total[5m])
rate(process_cpu_user_seconds_total[5m])

Dashboard

Finally the Dashboard looks as follows