Prometheus Observability Platform: Application metrics

When creating our own applications, we need to use a metrics library to generate the metrics and then inside our application functions increment said metrics. With Go for example, we can use the Prometheus library. The metrics will then be exposed to the /metrics endpoint. If our application is inside a Kubernetes cluster with a prometheus-operator, we can use a ServiceMonitor to scrape its metrics. If we don’t have such a possibility, we can instead set up an application to send metrics straight to our long-term storage solution. For VictoriaMetrics, we can use the github.com/VictoriaMetrics/metrics library to send the metrics to VictoriaMetrics. Remember to add authentication logic into the section pushing the metrics to the long-term storage, if necessary.

Demo

This example assumes that you have completed the following steps as the components from those are needed:

• Prometheus Observability Platform: Prometheus

Let's set up a hello-world Golang application in our cluster, and use ServiceMonitor to send its metrics to Prometheus.

First, we need to update our kube-prometheus-stack Helm deployment to pick up ServiceMonitor resources with a certain label attached. We need to pass the following value to our Helm chart:

prometheus:
  prometheusSpec:
    serviceMonitorSelector:
      matchExpressions:
      - key: app
        operator: Exists

I have converted that into json so that it can be passed to Helm:

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace prometheus --reuse-values --set-json 'prometheus.prometheusSpec.serviceMonitorSelector={"matchExpressions":[{"key":"app","operator":"Exists"}]}'

This will update our kube-prometheus-stack to pick up ServiceMonitor resources from any namespace, as long as they have an app label attached.

Next, we are going to create a namespace for our hello-world application which is a simple Golang application exposing metrics via the Prometheus module. We will borrow this already-made application, which has logic defined to increment a metric called hello_processed_total each time the page is loaded.

To create a namespace and a pod, we use the following commands:

kubectl create namespace hello-world

kubectl run hello-world --namespace=hello-world --image='okteto/hello-world:golang-metrics' --labels app=hello-world

Next, we need to create a service for the new pod:

cat <<'EOF' | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    app: hello-world
  name: hello-world
  namespace: hello-world
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: hello-world
  type: ClusterIP
EOF

Now we can test that our application is working by port-forwarding it. We can also check what the hello_processed_total metric looks like:

kubectl port-forward -n hello-world services/hello-world 9090:8080

Now navigate to http://localhost:9090 and http://localhost:9090/metrics. You should see a metric called hello_processed_total with a number attached. Each reload of the page will increment this number.

Next, we need to set up a ServiceMonitor to send these metrics to Prometheus:

cat <<'EOF' | kubectl create -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hello-world
  namespace: hello-world
  labels:
    app: hello-world
spec:
  selector:
    matchLabels:
      app: hello-world
  endpoints:
    - port: http
EOF

This ServiceMonitor will target services matching the label selector (app=hello-world) and will scrape the port called “http”.

Now, if we port-forward our Prometheus service, we should see a new service in the service discovery section:

kubectl port-forward -n prometheus services/kube-prometheus-stack-prometheus 9090:9090

Navigate to http://localhost:9090/service-discovery and you should see that there is a new service discovered with the name serviceMonitor/hello-world/hello-world/0 and it should show 1/1 active targets.

We can now query the hello_processed_total metric:

We have now achieved sending metrics from our custom app running in its own namespace into Prometheus.

Next part: Prometheus Observability Platform: Grafana