Centralised logging in microservices using Grafana stack and Fluent-bit

In this blog, we will explore how to set up a Grafana stack and Fluent-bit on Docker, alongside a Node.js application.

Introduction to the Stack:

The Grafana stack includes three main components: Grafana (an admin web portal), Loki (a datastore for logs), and Fluent-bit (a log collector). Here’s a brief overview of their purposes:

Fluent-bit: Fetches logs from the origin server, applies filters, and sends them to the datastore.

Loki: Stores logs with indexing and metadata.

Grafana: Analyzes, queries, and monitors your service via an Admin UI running on an individual system.

Additionally, the ELK stack (Elasticsearch, Logstash, Kibana) can be used for log aggregation and monitoring in microservices applications. Let’s compare these two stacks.

Elasticsearch:

Elasticsearch is a search engine tool built with Lucene. It stores unstructured data as JSON objects in its datastore, collected from Logstash, and visualizes logs through Kibana.

Elasticsearch indexes all contents provided by Logstash and stores them as documents. This makes all keys in the document searchable, requiring more storage space.

Loki:
Loki is a log aggregation tool that stores data as key-value pairs and includes labels with the log data.

Data Storage: Data in Loki is searchable by labels, resulting in lower indexing and more efficient storage.

What to use:
Loki: Ideal for storage efficiency and handling fewer log streams, as it uses less storage space and fetches logs based on labels.

Elasticsearch: Suitable for large datasets with detailed logs, as it indexes all fields and allows quick searchability.

Integration of the Grafana Stack with Docker

 fluent-bit:
    environment:
      LOG_LEVEL: debug
      LOKI_URL: http://loki:3100/loki/api/v1/push
    build:
      context: ./fluent-bit
      dockerfile: Dockerfile
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    networks:
      - backend
  loki:
    image: grafana/loki:latest
    expose:
      - "3100"
    networks:
      - backend
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_RENDERING_SERVER_URL: http://renderer:8081/render
      GF_RENDERING_CALLBACK_URL: http://grafana:3000/
      GF_LOG_FILTERS: rendering:debug
    networks:
      - backend
  renderer:
    image: grafana/grafana-image-renderer:latest
    expose:
      - "8081"
    environment:
      ENABLE_METRICS: "true"
    networks:
      - backend
networks:
  backend:
    driver: bridge

Here, we’re using the custom image of Fluent-bit as we’re using it for the Grafana stack. Grafana provides its Fluent-bit image for integration with Loki. You can check out the documentation here.

Create a fluent-bit folder at the root of your project, and within that folder, create a Dockerfile.

FROM grafana/fluent-bit-plugin-loki:latest
COPY ./conf/fluent.conf /fluent-bit/etc/fluent-bit.conf

Create a fluent-bit folder at the root of your project. Within that folder, create a Dockerfile, and a conf folder to hold the configuration files.

[INPUT]
    Name        forward
    Listen      0.0.0.0
    Port        24224
[Output]
    Name grafana-loki
    Match *
    Url ${LOKI_URL}
    RemoveKeys source
    Labels {job="fluent-bit"}
    LabelKeys container_name
    BatchWait 1s
    BatchSize 1001024
    LineFormat json
    LogLevel ${LOG_LEVEL}

We're passing LOKI_URL here as an environment variable.

To update the logging in the docker container update the docker-compose configuration for the server in your docker-compose.yml file.

 order-service:
    build:
      context: ./order-service
      dockerfile: Dockerfile
    restart: on-failure
    ports:
      - "9001:9001"
    env_file:
      - ./order/.env
    networks:
      - backend
    volumes:
      - ./order:/app/order
      - /app/order/node_modules
    logging:
      driver: fluentd
      options:
        fluentd-async: "true"
        fluentd-address: localhost:24224
        tag: order-service

In the Docker Compose configuration, the logging driver used is fluentd since Fluent-bit is part of the Fluentd ecosystem. The option fluentd-async: true enables an asynchronous connection with the Fluent-bit instance within the Docker environment, ensuring that the order service starts without errors. The tag option will be used in the Grafana UI to identify each service.

To start your microservice application, run the command docker-compose up in the terminal. This will start all Docker containers simultaneously.

According to the Docker Compose configuration file, the Grafana instance should be running on port 3000. Open Grafana in any browser by visiting http://localhost:3000. Use the default username admin and password admin to access the portal. You will be prompted to set your preferred password after logging in.

After login, it will show the dashboard screen.

Click on add a data source.

Click on Loki and enter URL: http://loki:3100 as we're using loki in docker. so we can use the docker container name to access the service in localhost.

Click on save & test, it will save the data source in grafana admin. and then click on explore to open the query dashboard.

Click on run query to show logs for every job that is available in the grafana stack. if you're looking for specific logs of any service. you can query logs from the log browser.

To learn more about the query language, you can check out the documentation provided by grafana on logQL.

Conclusion:
Adding centralized logging simplifies debugging and monitoring of a microservice application by providing a single UI to oversee all services. This approach becomes increasingly beneficial as the number of services in the system grows, making it challenging to monitor them individually on a cloud platform. Choosing the right logging stack can significantly improve the debugging process, especially when dealing with production application crashes.

Thanks for reading this. If you have any queries, feel free to email me at harsh.make1998@gmail.com.

Until next time!