DEV Community

Cover image for Monitoring the Kubernetes Nginx Ingress Controller with Prometheus and Grafana
Xavier Geerinck
Xavier Geerinck

Posted on

Monitoring the Kubernetes Nginx Ingress Controller with Prometheus and Grafana

cover

In a previous article I explained how we can set-up an Nginx Kubernetes Ingress Controller, but how can we now monitor this? This is what I would like to tackle in this article, on how we are able to utilize Prometheus and Grafana to start visualizing what is happening on our Ingress Controller.

Prerequisites

After following my previous article you will now have the following components running:

  • An Nginx Ingress controller deployed in the ingress-nginx namespace
  • A demo application that we can reach through a ingress route

If you do not have these 2 components running, I would recommend you to go check it out!

Setting up Prometheus

When we open the reference of ingress-nginx online we can see that it should be quite straightforward to install prometheus.

Simply run the following to deploy and configure the Prometheus Server:

kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/
Enter fullscreen mode Exit fullscreen mode

Once this is done (~2 min) you will see some output of everything that was created:

serviceaccount/prometheus-server created
role.rbac.authorization.k8s.io/prometheus-server created
rolebinding.rbac.authorization.k8s.io/prometheus-server created
configmap/prometheus-configuration-hct76d4c56 created
service/prometheus-server created
deployment.apps/prometheus-server created
Enter fullscreen mode Exit fullscreen mode

Now on to the "hard" part - reconfiguring our Nginx.

Note: at the time of writing (20-SEP-2020) the documentation was incomplete, making me spend 5h on the issue of reconfiguring nginx...

Reconfiguring Nginx

Reconfiguring Nginx to allow metrics to be sent to Prometheus might sound as an easy task, but actually it isn't... It appears that the official documentation is lacking on this point and that a recent pull request requires the ServiceMonitor to be used.

Installing the ServiceMonitor is a good option, but when checking this on the official repository it shows that this is still in "Beta" so it's not a really a good option to configure this one.

Luckily for us we can always check the values defined in our helm chart, which shows that we can still utilize the prometheus annotations.

Therefor we upgrade our Helm chart through the following command:

helm upgrade ingress-controller ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --set controller.metrics.enabled=true \
  --set-string controller.podAnnotations."prometheus\.io/scrape"="true" \
  --set-string controller.podAnnotations."prometheus\.io/port"="10254"
Enter fullscreen mode Exit fullscreen mode

When we validate this through the following command

helm get values ingress-controller --namespace ingress-nginx
Enter fullscreen mode Exit fullscreen mode

We can see that our values are now set as they should:

controller:
  metrics:
    enabled: true
    service:
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
Enter fullscreen mode Exit fullscreen mode

When I now open the Prometheus dashboard (remember: kubectl get nodes and get the external IP address, and open this with the prometheus port shown through kubectl get svc -A) and start typing ng it shows our metrics!

Installing Grafana

From here on, it's a smooth ride to the finish line! We can simply follow the official documentation and install grafana with:

kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/grafana/
Enter fullscreen mode Exit fullscreen mode

Once this is done, we do the same as for prometheus and open the grafana dashboard in our browser (kubectl get nodes; kubectl get svc -A) where we can use admin:admin as our credentials and run the following steps:

  1. Click "Add data source"
  2. Select "Prometheus"
  3. Enter the details (note: I used http://CLUSTER_IP_PROMETHEUS_SVC:9090)
  4. Left menu (hover over +) -> Dashboard
  5. Click "Import"
  6. Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/grafana/dashboards/nginx.json
  7. Click Import JSON
  8. Select the Prometheus data source
  9. Click "Import"

Conclusion

In this article we learned how we can now start monitoring our earlier application its ingress controller. The goal of this is to in a next post be able to auto-scale the created infrastructure based on the incoming requests!

Note: feel free to check the original post at my blog https://xaviergeerinck.com

Top comments (4)

Collapse
 
dflourusso profile image
Daniel Fernando Lourusso

Great article, thank's

Collapse
 
robincher profile image
Robin Cher

Great, thanks for the sharing. Manage to implement that.

Collapse
 
ajanth97 profile image
ajanth97

Great article ! I am not able to access the Prometheus dashboard because I cannot access the IP address of the prometheus-service. Could you please ellaborate on this ?

Collapse
 
ajanth97 profile image
ajanth97

I sorted it with a simple "kubectl port-forward"