DEV Community

Cover image for 🚀 Your Guide to Prometheus Monitoring on Kubernetes with Grafana
Lionel Tchami ♾️☁️ for AWS Community Builders

Posted on • Updated on • Originally published at softwaresennin.dev

🚀 Your Guide to Prometheus Monitoring on Kubernetes with Grafana

Introduction:

Hey fam🌟 In the fast-changing tech world of today, keeping an eye on the health of your apps has become the key to a smooth user experience. What do you know? Kubernetes is here to help you handle containers at scale as your trusted helper. But how do you keep track of all these bits of code that are flying around? Here come Prometheus and Grafana, a powerful pair that turns data into superhero insights.

K8S

Prometheus 📈

It is your measurements guru. It's an open-source wizard that not only gets data from your apps and services, but also adds some alerting magic. It's like having your own treasure chest full of info.

  • Open source monitoring tool
  • Out-of-the-box monitoring capabilities for the Kubernetes
  • It collects and stores metrics as time-series data, recording information with a timestamp
  • Works by pulling and collecting metrics from targets by scraping metrics HTTP endpoints.

Next, meet

Grafana 📊

It is your visual storyteller. It uses Prometheus data to create eye-catching visualizations. Think of your data as vibrant graphs that tell the story of your apps' performance and alert you to potential problems before they arise.

  • Open source visualization and analytics software.
  • It helps you to query, visualize, alert on, and even explore your metrics.

In this guide, we will set up Prometheus to watch your Kubernetes cluster and invite Grafana to the party. Whether you're an experienced Kubernetes user or just starting out, we've got you covered with everything you need to know to set up a rock-solid tracking system.

Grafana

So our Key Components are

Key components:

  1. Prometheus server — Processes and stores all your metrics data
  2. Alert Manager — The manager sends alerts to any systems/channels
  3. Grafana — Visualize all scraped data in your UI

go
Let's Go!!!!!!!!!!!!!!!!!

Installing

There are many ways to setup Prometheus and Grafana. Let us look at some of them:

  1. Set up your Prometheus and Grafana configuration files and run them in the correct order.

  2. Prometheus Operator: Used to streamline and automate the administration of your Prometheus monitoring stack in your Kubernetes environment.

  3. Helm Chart (Recommended) : Using Helm Chart, set up and Prometheus Operator which includes Grafana.

Why Helm ?

Helm is a package manager for Kubernetes. In other words, it simplifies the setting up and installation of all the components of an installation, all in one command. It is recommended to use Helm as it will take care of all the config steps and you would not miss any.

Helm has three significant benefits that it adds to the process of app deployments to the Kubernetes cluster:

  • Speed of Deployment – It helps us speed up our speed of app deployments with a single command.
  • Using prebuilt application configurations – Whatever the config that you need for your infrastructure, someone else already has a prebuilt application configurations that can be used.
  • Easy rollbacks – Last but not least, Helm makes it more easy for us to upgrade and rollback the versions of our apps.

So let us look at our prerequisites

Pre-requisites

  1. Setup Kubernetes (Using Kubeadm)
  2. Install Helm Package manager
  3. Download the Helm charts for setting up Prometheus and Grafana

Let us Get Started!

Creating our Infrastructure

We will be setting up our Kubernetes cluster on AWS, using Kubeadm. I will be using two (2) t2.medium ec2 instances for this installation.

ec2

Installing Kubernetes using Kubeadm

Step 1

We will ssh into the 2 VMs (the one we will use as MASTER & WORKER node) and run some commands there to configure the environment.

apt-get update -y  

apt-get install docker.io -y  

service docker restart  

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -  

echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" >/etc/apt/sources.list.d/kubernetes.list  

apt-get update
Enter fullscreen mode Exit fullscreen mode

Step 2

Now we need to configure the MASTER NODE. So now we will ssh into it and run some commands

kubeadm init --pod-network-cidr=192.168.0.0/16  

# If above one fails then run below command  

kubeadm token create --print-join-command
Enter fullscreen mode Exit fullscreen mode

Step 3

Let us proceed with our installation in the MASTER NODE

mkdir -p $HOME/.kube  

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config  

sudo chown $(id -u):$(id -g) $HOME/.kube/config
Enter fullscreen mode Exit fullscreen mode

Step 4

Next, we need to run some kubectl commands on our Master Node to complete the installation

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/calico.yaml  


kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.49.0/deploy/static/provider/baremetal/deploy.yaml  

(after that check "kubectl get nodes" command on master node you can see the worker node configured with master)
Enter fullscreen mode Exit fullscreen mode

Let us look at what it looks like now

Here

Installing Helm

Now we can install Helm on our master node. For that, you can run these commands.

curl -fsSL -o get_helm.sh \ https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 

chmod 700 get_helm.sh 

./get_helm.sh
Enter fullscreen mode Exit fullscreen mode

pic

Setting up our Monitoring Environment

Now that we have installed Helm Package Manager we need to add Helm Stable Charts for our local machine.

helm repo add stable https://charts.helm.sh/stable
Enter fullscreen mode Exit fullscreen mode

stable

Next, we will be adding the Helm repository of prometheus to our local machine

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Enter fullscreen mode Exit fullscreen mode

Next we need to create the namespace where we will install prometheus

kubectl create namespace prometheus
Enter fullscreen mode Exit fullscreen mode

We can now go ahead and install our kube-prometheus stack. The kube-stack-prometheus will be installed with Helm, with a grafana deployment embedded.

helm install stable prometheus-community/kube-prometheus-stack --version 48.3.1 -n prometheus
Enter fullscreen mode Exit fullscreen mode

-n is added to specify the namespace where you want the installation to be done.

Having installed it ... this is the screen we will have next

prom

Great Job!!!!! It is successfully installed. Now let us go ahead and check our pod resources

kubectl get pods -n prometheus
Enter fullscreen mode Exit fullscreen mode

kube

kubectl get svc -n prometheus
Enter fullscreen mode Exit fullscreen mode

kubee

These all show that broth Prometheus and Grafana have been successfully installed.

Enabling external access to our Infrastructure

When a Kubernetes cluster has been created, Cluster IP is usually created. However, for us to be able to enable external access, we need to have a LoadBalancer or NodePort service.

For that, we will therefore need to edit the Prometheus and Grafana service. Run the following command

kubectl edit svc stable-kube-prometheus-sta-prometheus -n prometheus
Enter fullscreen mode Exit fullscreen mode

pro

kubectl edit svc stable-grafana -n prometheus
Enter fullscreen mode Exit fullscreen mode

gra

Once you finish editing both services, make sure that you close and save the file -> Escape :wq

Now we should have 2 loadbalancers in our Cluster. Let us check that

load

Now we can access Grafana. So copy the Loadbalancer link.

graff

grafana

Configuring our Grafana server

Grafana dashboards give you instant visual insights, making complex data easier to understand. They also let you watch in real time and help you make decisions based on data.

Discovering the password 🔍

To login to your grafana account, we will use the default username and password. The default user is admin and the password can be found with the following command

kubectl get secrets
Enter fullscreen mode Exit fullscreen mode

You will be see different secrets as below

here

Mine is stable-grafana. Replace that with yours.

kubectl get secret stable-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
Enter fullscreen mode Exit fullscreen mode

Please note that stable-grafana is the name of the grafana secret on my deployment. Yours could be different.

We can create different kinds of dashboards, depending on our needs.

Grafana dashboards are very easy to create. Let us go through the steps to create our dashboard. In your Grafana server, go ahead and create our dashboards.

1. Creating Kubernetes Monitoring Dashboard

  • Click + button on left panel and select ‘Import’
  • Enter 12740 dashboard id under Grafana.com Dashboard
  • Click ‘Load’
  • Select ‘Prometheus’ as the endpoint under Prometheus Data Sources drop down.
  • Click ‘Import’

This DASHBOARD will show monitoring dashboard for all the nodes in our cluster.

graf

2. Creating Kubernetes Cluster Monitoring Dashboard

  • Click + button on left panel and select ‘Import’
  • Enter 3119 dashboard id under Grafana.com Dashboard

ret

  • Click ‘Load’

  • Select ‘Prometheus’ as the endpoint under Prometheus Data Sources drop down.
    grafff

  • Click ‘Import’

This DASHBOARD will show monitoring dashboard for all the nodes in our cluster.

frt

3. Creating POD Monitoring Dashboard

  • Click + button on left panel and select ‘Import’
  • Enter 6417 dashboard id under Grafana.com Dashboard
  • Click ‘Load’
  • Select ‘Prometheus’ as the endpoint under Prometheus Data Sources drop down.
  • Click ‘Import’

pod

Congratulations!!!!!!!!!

yay


Conclusion

Your journey to master Prometheus Monitoring on Kubernetes using Grafana is complete! 🚀 You can effortlessly gather, visualise, and analyse metrics from your dynamic Kubernetes system thanks to this hands-on tutorial.

By combining Prometheus' data collection strength with Grafana's stunning dashboards, you can confidently sail containerized application into the seas. From resource utilization to anomaly detection and alert setup, you've learned key skills for monitoring system health and performance.

Prometheus and Grafana help you optimise, debug, and innovate with Kubernetes and monitoring. Hope your monitoring trip is informative and your applications sparkle! 🌟🌐📊

Top comments (6)

Collapse
 
nayanb profile image
Nayan Biswas

So good 👍🏻

Collapse
 
softwaresennin profile image
Lionel Tchami ♾️☁️

awesome! Thanks so much for commenting.

Collapse
 
adesoji1 profile image
Adesoji1

Nice work. Thank you for this

Collapse
 
softwaresennin profile image
Lionel Tchami ♾️☁️

Thanks so much for commenting, I am glad it is useful to you. Cheers!

Collapse
 
respect17 profile image
Kudzai Murimi

Thanks a lot!

Collapse
 
softwaresennin profile image
Lionel Tchami ♾️☁️

You are very welcome. I am so glad this is helpful to you.