DEV Community

eiymba
eiymba

Posted on • Updated on

Adding Loki From DigitalOcean's Loki 1-Click Installer To Grafana In Your DigitalOcean 1-Click Prometheus Monitoring Stack

TL;DR

🍬 Set your Loki HTTP URL to:

http://loki.loki-stack.svc.cluster.local:3100
Enter fullscreen mode Exit fullscreen mode

Contents

Note: This guide assumes you do not have another Loki service running, and that the Loki service installed by DigitalOcean is called loki.

Prerequisites

  • A DigitalOcean account.
  • A kubernetes cluster set up on DigitalOcean.
  • Prometheus Monitoring Stack already installed as a 1-click application.
  • Loki stack already installed as a 1-click application.
  • Administrator privileges in the Grafana dashboard which exists in the kube-prometheus-stack namespace.
  • DNS resolver (comes as standard with DigitalOcean's kubernetes service).
  • You're using the defaults of the 1-click installers.
  • You can port forward the Grafana service using kubectl (needs to be installed independently on your local machine, and connected to your cluster) to access your Grafana dashboard, e.g:

    kubectl port-forward svc/kube-prometheus-stack-grafana -n kube-prometheus-stack 3000:80
    
    • Note: This example assumes port 3000 is available on your machine. If it's already in use, change the 3000:80 part of the command to something else, followed by :80, e.g. 5000:80.
    • You can then access your Grafana dashboard by navigating to: http://localhost:3000 in your browser, or whatever you've set your local port as.

Introduction

DigitalOcean allows you to install a Prometheus monitoring stack using a 1-click installation option, alongside a Loki stack.

The problem is, you end up with two sets of Grafana dashboards that are disjoint from each other; one in the kube-prometheus-stack namespace created by the Prometheus Monitoring Stack's 1-click installer, and one in the loki-stack namespace created by the Loki 1-click installer.

My goal was to have the Grafana dashboard in the kube-prometheus-stack display logs from the Loki service running under the loki-stack namespace.

Here's how I did it...

Add Loki As A Data Source

  1. From your browser, navigate to the Grafana dashboard which exists in the kube-prometheus-stack namespace (see prerequisites if you're not sure how). On the side navigation panel of the Grafana UI, hover over ⚙ Configuration to reveal a sub menu. Inside this sub menu, select Data sources. Adding a data source

  2. Click on the Add data source button. Clicking on add data source

  3. Find Loki as a data source and click Select. Note: You may need to search for Loki in the search bar to find it. Selecting Loki as a data source

  4. The only field you need to enter is the URL field. Use the following value:

    http://loki.loki-stack.svc.cluster.local:3100
    


    Afterwards, click Save & test to apply your settings. (Didn't work?) Loki data source configuration Note: If you have a previous Loki data source added, rename this new source aptly, then proceed with your chosen name for step 8.

  5. If you already have a dashboard created, skip this and go to step 6.
    Create a new dashboard by hovering over the ➕ Create menu item on the left navigation bar. In the sub menu revealed, select Dashboard. Creating a new dashboard

  6. Click the Add panel icon on the top right of the dashboard. Add panel button

  7. Select Add an empty panel. Add an empty panel button

  8. Configure the data source to be Loki (or the name given to the data source in step 4). Data source set to Loki

  9. Click on the visualisation type to change it from the default (in this case, time series), to Logs.
    i. Visualisation option
    ii. Logs option

  10. Enter a query to read the stdout of a pod, for example:

    {app="loki"}
    


    An example query

  11. (Optional) Consider reducing the number of data points being requested to something more modest, e.g. 100. Query options

  12. (Optional) If your logs are timestamped, you may be able to group them and format the timestamp. Log formatting options

  13. Apply your changes. Applying changes

If all goes well, you should be able to see your desired logs using Loki, as installed by DigitalOcean's 1-click installer. 🥳 A dashboard with Loki

How Does It Work?

The Loki 1-Click installer creates a service called loki in the loki-stack namespace, with an endpoint which the DNS resolver can interpret using the following syntax:

<service_name>.<namespace>.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

Loki is running inside a pod where port 3100 is being used. Thus, the complete syntax is:

<service_name>.<namespace>.svc.cluster.local:<port>
Enter fullscreen mode Exit fullscreen mode

Troubleshooting

I'm not an expert. 🤷‍♂️

502: Bad Gateway

Translation: Loki is not accessible using the URL you've provided. That probably means your installation is different to what the 1-click installer provides, or DigitalOcean has changed the way it installs Loki after this article was written.

Data source connected, but no labels received

The most likely culprits are:

  • You're pointing to the wrong service. Be sure to double check the name of the service using kubectl get svc -n loki-stack, and verify that a service with the name loki exists.
  • The Loki service has a different endpoint. Verify that the endpoint with the name loki exists using kubectl get endpoints -n loki-stack.
  • A firewall policy. GG well met. ☠ Just kidding, but I'm no expert on kubernetes or its network policy configurations.

Discussion (0)