DEV Community

Cover image for Loki: Effective Logging and Log Aggregation with Grafana
Grzegorz Piechnik
Grzegorz Piechnik

Posted on

Loki: Effective Logging and Log Aggregation with Grafana

If you are a programmer or application administrator, you surely understand how important logs are in the infrastructure. Through them, you can monitor what's happening under the hood of your application, detect issues, and analyze system performance and behavior in real time. However, what happens when the volume of logs increases and analysis becomes more complex?

Introducing Loki

This is where Loki comes in - an advanced log aggregation system developed by Grafana Labs. Loki aims to simplify effective and user-friendly collection and storage of logs.

Loki is designed to resemble Prometheus, allowing for quick and intuitive data filtering. Compared to Prometheus, the creators of Grafana decided to forego storing metrics in favor of storing logs themselves. In my opinion this decision stemmed from the desire to break monitoring tools into smaller, more modular parts. Instead of creating an all-in-one tool, they chose an approach based on several smaller tools. This allows for utilizing only the features that are relevant to a specific project, providing greater flexibility and control over the project.

A key feature of Loki is that it indexes only labels and metadata for each log message. This means that the full log contents are not stored. On one hand, this improves the tool's performance and reduces maintenance costs. On the other hand, this approach might somewhat limit analysis. However, it's worth keeping in mind that everything comes at a cost.

Loki 1

Loki's Technological Stack

In practice, Loki consists of three key components:

  1. Promtail - This is an agent that operates in a "push" configuration rather than "pull." This means its task is to acquire logs and send them to Loki, instead of waiting for queries from Loki. Multiple Promtail agents can run on a single machine, collecting and forwarding logs.
  2. Loki - Loki is responsible for storing our logs. It indexes only the labels and metadata of each message, enabling efficient filtering and searching. Loki collects and provides log data.
  3. Grafana (or another visualization system) - This component is used to interact with Loki. Grafana queries Loki to perform filtering and select the desired results from the logs. These data can then be visualized as charts or other graphics.

The operational flow of this tool can be simplified using the following diagram:

Loki 2

With the theoretical understanding in place, let's move on to the practical aspect.

Deploying Loki

To demonstrate the tool's functionality, we will use the Docker platform. Specifically, we will use a pre-made docker-compose file available here, while making certain modifications. This file will allow us to sequentially launch Grafana, Loki, Promtail, and an Nginx server.

Here's the content of the docker-compose file after making certain changes:

version: '3.4'


        container_name: nginx-app
        image: nginx
        logging: "promtail"
        logging_jobname: "containerlogs"
            - 8080:80
            - app

        image: grafana/grafana:latest
            - 3000:3000
            - ./config/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yaml
            - GF_AUTH_ANONYMOUS_ENABLED=true
            - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
            - GF_AUTH_DISABLE_LOGIN_FORM=true
            - app

        image: grafana/loki:latest
            - 3100:3100
        command: -config.file=/etc/loki/local-config.yaml
            - app

        image: grafana/promtail:latest
        container_name: promtail
            - ./config/promtail.yaml:/etc/promtail/docker-config.yaml
            - /var/lib/docker/containers:/var/lib/docker/containers:ro
            - /var/run/docker.sock:/var/run/docker.sock
        command: -config.file=/etc/promtail/docker-config.yaml
            - loki
            - app
Enter fullscreen mode Exit fullscreen mode

Additionally, we have created two configuration files. The first of them is responsible for the data source in Grafana, while the second one governs Promtail settings.

File grafana-datasources.yml:

apiVersion: 1

    - name: Loki
        type: loki
        access: proxy
        url: http://loki:3100
        version: 1
        editable: false
        isDefault: true
Enter fullscreen mode Exit fullscreen mode

File promtail.yaml:

    http_listen_port: 9080
    grpc_listen_port: 0

    filename: /tmp/positions.yaml

    - url: http://loki:3100/loki/api/v1/push

    - job_name: flog_scrape
            - host: unix:///var/run/docker.sock
            refresh_interval: 5s
            - name: label
            values: ["logging=promtail"]
            - source_labels: ['__meta_docker_container_name']
            regex: '/(.*)'
            target_label: 'container'
            - source_labels: ['__meta_docker_container_log_stream']
            target_label: 'logstream'
            - source_labels: ['__meta_docker_container_label_logging_jobname']
            target_label: 'job'
Enter fullscreen mode Exit fullscreen mode

After running the docker-compose up command, containers for Grafana, Loki, Promtail, and Nginx will be created. Upon accessing the local address with port 3000, you will have the ability to create visualizations of your logs.

Loki 3

As we can see, with this approach, we can retrieve logs from a container with a specific name, in our case, it will be nginx-app.

Types of Logs

There are several different types of logs, including:

  1. Event Log: An event log is a high-level record that logs information about network traffic and interactions, such as login attempts, failed authentication attempts, and application-related events.
  2. Server Log: A server log is a text file containing records of actions related to a specific server over a defined period of time.
  3. System Log (syslog): A system log, also known as syslog, is a record of system events. It includes messages about system startup, changes to the system, unexpected shutdowns, errors, warnings, and other important processes. Windows, Linux, and macOS operating systems generate syslog messages.
  4. Authorization Logs and Access Logs: Authorization logs and access logs contain lists of individuals or bots who accessed specific applications or files.
  5. Change Logs: Change logs provide a chronological list of modifications made to an application or file.
  6. Availability Logs: Availability logs track system performance, uptime, and availability.
  7. Resource Logs: Resource logs provide information about connectivity issues and capacity limitations.
  8. Threat Logs: Threat logs contain information about system traffic, files, or applications that match a predefined security profile within a firewall.

The method of logging data will depend on the application architecture and programming language. For example, for applications written in Django or Flask, you can use a library called python-logging-loki. Adding logs to Loki could look like the following:

import logging
import logging_loki

handler = logging_loki.LokiHandler(
    tags={"application": "my-app"},
    auth=("username", "password"),

logger = logging.getLogger("my-logger")
    "Something happened", 
    extra={"tags": {"service": "my-service"}},
Enter fullscreen mode Exit fullscreen mode


In the discussed article, we learned about a tool called Loki, created by Grafana Labs, which is useful for effective log management in programming environments. We discovered that Loki is a system designed for collecting and processing various types of logs. Through its innovative "logs as queries" (log-as-data) approach, Loki enables efficient storage of vast amounts of logs while minimizing resource consumption.

Additionally, we discussed various types of logs that can be collected and analyzed using Loki.

Top comments (0)