Adding Loki From DigitalOcean's Loki 1-Click Installer To Grafana In Your DigitalOcean 1-Click Prometheus Monitoring Stack

TL;DR

🍬 Set your Loki HTTP URL to:

http://loki.loki-stack.svc.cluster.local:3100

Contents

• Prerequisites
• Introduction
• Add Loki As A Data Source
• How Does It Work?
• Troubleshooting
  • 502: Bad Gateway
  • Data source connected, but no labels received

Note: This guide assumes you do not have another Loki service running, and that the Loki service installed by DigitalOcean is called loki.

Prerequisites

• A DigitalOcean account.
• A kubernetes cluster set up on DigitalOcean.
• Prometheus Monitoring Stack already installed as a 1-click application.
• Loki stack already installed as a 1-click application.
• Administrator privileges in the Grafana dashboard which exists in the kube-prometheus-stack namespace.
• DNS resolver (comes as standard with DigitalOcean's kubernetes service).
• You're using the defaults of the 1-click installers.

• You can port forward the Grafana service using kubectl (needs to be installed independently on your local machine, and connected to your cluster) to access your Grafana dashboard, e.g:
  

  kubectl port-forward svc/kube-prometheus-stack-grafana -n kube-prometheus-stack 3000:80
  

  • Note: This example assumes port 3000 is available on your machine. If it's already in use, change the 3000:80 part of the command to something else, followed by :80, e.g. 5000:80.
  • You can then access your Grafana dashboard by navigating to: http://localhost:3000 in your browser, or whatever you've set your local port as.

Introduction

DigitalOcean allows you to install a Prometheus monitoring stack using a 1-click installation option, alongside a Loki stack.

The problem is, you end up with two sets of Grafana dashboards that are disjoint from each other; one in the kube-prometheus-stack namespace created by the Prometheus Monitoring Stack's 1-click installer, and one in the loki-stack namespace created by the Loki 1-click installer.

My goal was to have the Grafana dashboard in the kube-prometheus-stack display logs from the Loki service running under the loki-stack namespace.

Here's how I did it...

Add Loki As A Data Source

1. From your browser, navigate to the Grafana dashboard which exists in the kube-prometheus-stack namespace (see prerequisites if you're not sure how). On the side navigation panel of the Grafana UI, hover over ⚙ Configuration to reveal a sub menu. Inside this sub menu, select Data sources.

2. Click on the Add data source button.

3. Find Loki as a data source and click Select. Note: You may need to search for Loki in the search bar to find it.

4. The only field you need to enter is the URL field. Use the following value:
   

   http://loki.loki-stack.svc.cluster.local:3100
   

   
   Afterwards, click Save & test to apply your settings. (Didn't work?) Note: If you have a previous Loki data source added, rename this new source aptly, then proceed with your chosen name for step 8.

5. If you already have a dashboard created, skip this and go to step 6.
   Create a new dashboard by hovering over the ➕ Create menu item on the left navigation bar. In the sub menu revealed, select Dashboard.

6. Click the Add panel icon on the top right of the dashboard.

7. Select Add an empty panel.

8. Configure the data source to be Loki (or the name given to the data source in step 4).

9. Click on the visualisation type to change it from the default (in this case, time series), to Logs.
   i.
   ii.

10. Enter a query to read the stdout of a pod, for example:
    

    {app="loki"}
    

    
    

11. (Optional) Consider reducing the number of data points being requested to something more modest, e.g. 100.

12. (Optional) If your logs are timestamped, you may be able to group them and format the timestamp.

13. Apply your changes.

If all goes well, you should be able to see your desired logs using Loki, as installed by DigitalOcean's 1-click installer. 🥳

How Does It Work?

The Loki 1-Click installer creates a service called loki in the loki-stack namespace, with an endpoint which the DNS resolver can interpret using the following syntax:

<service_name>.<namespace>.svc.cluster.local

Loki is running inside a pod where port 3100 is being used. Thus, the complete syntax is:

<service_name>.<namespace>.svc.cluster.local:<port>

Troubleshooting

I'm not an expert. 🤷‍♂️

502: Bad Gateway

Translation: Loki is not accessible using the URL you've provided. That probably means your installation is different to what the 1-click installer provides, or DigitalOcean has changed the way it installs Loki after this article was written.

Data source connected, but no labels received

The most likely culprits are:

• You're pointing to the wrong service. Be sure to double check the name of the service using kubectl get svc -n loki-stack, and verify that a service with the name loki exists.
• The Loki service has a different endpoint. Verify that the endpoint with the name loki exists using kubectl get endpoints -n loki-stack.
• A firewall policy. GG well met. ☠ Just kidding, but I'm no expert on kubernetes or its network policy configurations.