Loki: Effective Logging and Log Aggregation with Grafana

If you are a programmer or application administrator, you surely understand how important logs are in the infrastructure. Through them, you can monitor what's happening under the hood of your application, detect issues, and analyze system performance and behavior in real time. However, what happens when the volume of logs increases and analysis becomes more complex?

Introducing Loki

This is where Loki comes in - an advanced log aggregation system developed by Grafana Labs. Loki aims to simplify effective and user-friendly collection and storage of logs.

Loki is designed to resemble Prometheus, allowing for quick and intuitive data filtering. Compared to Prometheus, the creators of Grafana decided to forego storing metrics in favor of storing logs themselves. In my opinion this decision stemmed from the desire to break monitoring tools into smaller, more modular parts. Instead of creating an all-in-one tool, they chose an approach based on several smaller tools. This allows for utilizing only the features that are relevant to a specific project, providing greater flexibility and control over the project.

A key feature of Loki is that it indexes only labels and metadata for each log message. This means that the full log contents are not stored. On one hand, this improves the tool's performance and reduces maintenance costs. On the other hand, this approach might somewhat limit analysis. However, it's worth keeping in mind that everything comes at a cost.

Loki's Technological Stack

In practice, Loki consists of three key components:

1. Promtail - This is an agent that operates in a "push" configuration rather than "pull." This means its task is to acquire logs and send them to Loki, instead of waiting for queries from Loki. Multiple Promtail agents can run on a single machine, collecting and forwarding logs.
2. Loki - Loki is responsible for storing our logs. It indexes only the labels and metadata of each message, enabling efficient filtering and searching. Loki collects and provides log data.
3. Grafana (or another visualization system) - This component is used to interact with Loki. Grafana queries Loki to perform filtering and select the desired results from the logs. These data can then be visualized as charts or other graphics.

The operational flow of this tool can be simplified using the following diagram:

With the theoretical understanding in place, let's move on to the practical aspect.

Deploying Loki

To demonstrate the tool's functionality, we will use the Docker platform. Specifically, we will use a pre-made docker-compose file available here, while making certain modifications. This file will allow us to sequentially launch Grafana, Loki, Promtail, and an Nginx server.

Here's the content of the docker-compose file after making certain changes:

version: '3.4'

networks:
    app:

services:
    nginx-app:
        container_name: nginx-app
        image: nginx
        labels:
        logging: "promtail"
        logging_jobname: "containerlogs"
        ports:
            - 8080:80
        networks:
            - app

    grafana:
        image: grafana/grafana:latest
        ports:
            - 3000:3000
        volumes:
            - ./config/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yaml
        environment:
            - GF_AUTH_ANONYMOUS_ENABLED=true
            - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
            - GF_AUTH_DISABLE_LOGIN_FORM=true
        networks:
            - app

    loki:
        image: grafana/loki:latest
        ports:
            - 3100:3100
        command: -config.file=/etc/loki/local-config.yaml
        networks:
            - app

    promtail:
        image: grafana/promtail:latest
        container_name: promtail
        volumes:
            - ./config/promtail.yaml:/etc/promtail/docker-config.yaml
            - /var/lib/docker/containers:/var/lib/docker/containers:ro
            - /var/run/docker.sock:/var/run/docker.sock
        command: -config.file=/etc/promtail/docker-config.yaml
        depends_on:
            - loki
        networks:
            - app

Additionally, we have created two configuration files. The first of them is responsible for the data source in Grafana, while the second one governs Promtail settings.

File grafana-datasources.yml:

apiVersion: 1

datasources:
    - name: Loki
        type: loki
        access: proxy
        url: http://loki:3100
        version: 1
        editable: false
        isDefault: true

File promtail.yaml:

server:
    http_listen_port: 9080
    grpc_listen_port: 0

positions:
    filename: /tmp/positions.yaml

clients:
    - url: http://loki:3100/loki/api/v1/push

scrape_configs:
    - job_name: flog_scrape
        docker_sd_configs:
            - host: unix:///var/run/docker.sock
            refresh_interval: 5s
            filters:
            - name: label
            values: ["logging=promtail"]
        relabel_configs:
            - source_labels: ['__meta_docker_container_name']
            regex: '/(.*)'
            target_label: 'container'
            - source_labels: ['__meta_docker_container_log_stream']
            target_label: 'logstream'
            - source_labels: ['__meta_docker_container_label_logging_jobname']
            target_label: 'job'

After running the docker-compose up command, containers for Grafana, Loki, Promtail, and Nginx will be created. Upon accessing the local address with port 3000, you will have the ability to create visualizations of your logs.

As we can see, with this approach, we can retrieve logs from a container with a specific name, in our case, it will be nginx-app.

Types of Logs

There are several different types of logs, including:

1. Event Log: An event log is a high-level record that logs information about network traffic and interactions, such as login attempts, failed authentication attempts, and application-related events.
2. Server Log: A server log is a text file containing records of actions related to a specific server over a defined period of time.
3. System Log (syslog): A system log, also known as syslog, is a record of system events. It includes messages about system startup, changes to the system, unexpected shutdowns, errors, warnings, and other important processes. Windows, Linux, and macOS operating systems generate syslog messages.
4. Authorization Logs and Access Logs: Authorization logs and access logs contain lists of individuals or bots who accessed specific applications or files.
5. Change Logs: Change logs provide a chronological list of modifications made to an application or file.
6. Availability Logs: Availability logs track system performance, uptime, and availability.
7. Resource Logs: Resource logs provide information about connectivity issues and capacity limitations.
8. Threat Logs: Threat logs contain information about system traffic, files, or applications that match a predefined security profile within a firewall.

The method of logging data will depend on the application architecture and programming language. For example, for applications written in Django or Flask, you can use a library called python-logging-loki. Adding logs to Loki could look like the following:

import logging
import logging_loki

handler = logging_loki.LokiHandler(
    url="https://my-loki-instance/loki/api/v1/push", 
    tags={"application": "my-app"},
    auth=("username", "password"),
    version="1",
)

logger = logging.getLogger("my-logger")
logger.addHandler(handler)
logger.error(
    "Something happened", 
    extra={"tags": {"service": "my-service"}},
)

Summary

In the discussed article, we learned about a tool called Loki, created by Grafana Labs, which is useful for effective log management in programming environments. We discovered that Loki is a system designed for collecting and processing various types of logs. Through its innovative "logs as queries" (log-as-data) approach, Loki enables efficient storage of vast amounts of logs while minimizing resource consumption.

Additionally, we discussed various types of logs that can be collected and analyzed using Loki.