DEV Community

Kaito Ii
Kaito Ii

Posted on

Remote writing your Prometheus metrics to Grafana Cloud

Setup Grafana Cloud

Register to Grafana Cloud.

Create API Key for Grafana Cloud

We will create an API Key to allow our local Prometheus to remote write to Grafana Cloud.

Grafana Cloud Portal

  • Head over to your Grafana Cloud Portal and select Send Metrics on Prometheus.

Prometheus Configs

  • If you scroll above, you should see the section for API Key. Click on Generate now and create an API Key with the Role MetricsPublisher.

  • Copy the Prometheus config and save it locally as values.yaml as we will use it to install our Prometheus Helm Chart. The url and username should be unique for every user.
    The password in both snippet should be filled with your API key.

Local Setup

I used Rancher Desktop but most of Kubernetes distribution should work similarly. We will deploy Prometheus using kube-prometheus-stack with Helm.

  • Add Helm repository
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
Enter fullscreen mode Exit fullscreen mode

Before installing Prometheus, we will tweak the values.yaml we created earlier. We will not directly paste the username and password(API Key) to the file. Instead, we will create a secret and reference that from Prometheus Config.

$ kubectl create secret generic grafanacloudsecret \ 
--from-literal=username=<YOURUSERNAME> \
--from-literal=password='<YOURPASSWORD>' 
Enter fullscreen mode Exit fullscreen mode

Now, edit the values.yaml. Here is the edited version. Notice that the username and password is referencing the secret. Also I have added extra parameter which will attach extra labels to the metrics sent from this local cluster.

prometheus:
  prometheusSpec:
    remoteWrite:
    - url: "https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push"
      basicAuth:
          username:
            name: grafanacloudsecret
            key: username
          password:
            name: grafanacloudsecret
            key: password
    replicaExternalLabelName: "__replica__"
    externalLabels: {cluster: "mylocalcluster"}

Enter fullscreen mode Exit fullscreen mode

Now, let's install our Prometheus.

$ helm install prometheus prometheus-community/kube-prometheus-stack -n prometheus --create-namespace -f values.yaml

Enter fullscreen mode Exit fullscreen mode

Check the metrics on Grafana Cloud

Go back to your Grafana Cloud and open the Grafana Cloud Billing/Usage. If you wait awhile, we should see some metrics being ingested in Grafana Cloud.

Grafana Cloud Usage

On the left pane, Go to Explore. In the Metrics browser, type in container_memory_usage_bytes. We can see the memory usage in my local Kubernetes cluster. Notice that there is an extra label named cluster: "mylocalcluster" which we added in our values.yaml. This can be used as an identifier for where this data is coming from.

Prometheus metrics

Reference

Discussion (0)