In this blog, we will see how to create a Grafana stack and fluent-bit on docker along with a nodejs application.
Introduction to the stack:
Grafana stack includes - Grafana (admin web portal), Loki (datastore for logs), and fluent-bit (logs collector). purpose of fluent-bit is to fetch logs from the origin server, add filters on the logs and send them to the data store. purpose of Loki is to store logs with indexing and metadata on a system. purpose of Grafana is to get analyze/query/monitor your service by providing Admin UI running on an individual system.
we can also use ELK (elastic search, logstash, kibana) stack for log aggregation and monitoring for a microservices application. let's check out the difference between both stacks.
Elasticsearch:
Elasticsearch is a search engine tool and it is built with Lucene. elastic search stored unstructured data as a JSON object in its data store which was collected from logstash
and kibana let the user visualize logs on the admin portal.
elastic search stores data in the following manner. it will indexes all the contents that were provided by logstash and store the document. it will make elastic search searchable from every key in the document thus it required more space for storage.
Loki:
Loki is a log aggregation tool that will also store data as a key-value pair. but it also stores labels with log data. so data from Loki is searchable with the use of labels thus it creates low indexes in the data storage system and it will make it more storage efficient.
What to use:
In terms of storage efficiency and fewer log streams loki is a good choice as it doesn't use much storage space and can fetch logs streams based on labels. but, if you have a large dataset with logs for example, you have metadata and other extra fields for your logs then loki will take much time to fetch log streams as it will create multiple labels and create multiple log streams on your data. in this case, the elastic search can be a good option for storing this kind of log data as it will create indexes for all the fields that you store in the data store.
Let's begin with the integration of the Grafana stack with docker.
fluent-bit:
environment:
LOG_LEVEL: debug
LOKI_URL: http://loki:3100/loki/api/v1/push
build:
context: ./fluent-bit
dockerfile: Dockerfile
ports:
- "24224:24224"
- "24224:24224/udp"
networks:
- backend
loki:
image: grafana/loki:latest
expose:
- "3100"
networks:
- backend
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
GF_RENDERING_SERVER_URL: http://renderer:8081/render
GF_RENDERING_CALLBACK_URL: http://grafana:3000/
GF_LOG_FILTERS: rendering:debug
networks:
- backend
renderer:
image: grafana/grafana-image-renderer:latest
expose:
- "8081"
environment:
ENABLE_METRICS: "true"
networks:
- backend
networks:
backend:
driver: bridge
Here, we're using the custom image of fluent-bit as we're using it for the Grafana stack. Grafana provides its fluent bit image for the integration with loki. you can check out the documentation here.
Create fluent-bit
folder on the root of your project, and in that folder create Dockerfile
.
FROM grafana/fluent-bit-plugin-loki:latest
COPY ./conf/fluent.conf /fluent-bit/etc/fluent-bit.conf
We're passing the configuration to the docker image. create a configuration in a fluent-bit folder. create conf
folder and in that folder create file fluent-bit.conf
[INPUT]
Name forward
Listen 0.0.0.0
Port 24224
[Output]
Name grafana-loki
Match *
Url ${LOKI_URL}
RemoveKeys source
Labels {job="fluent-bit"}
LabelKeys container_name
BatchWait 1s
BatchSize 1001024
LineFormat json
LogLevel ${LOG_LEVEL}
We're passing LOKI_URL
here as an environment variable.
To update the logging
in the docker container update the docker-compose configuration for the server in your docker-compose.yml
file.
order:
build:
context: ./order
dockerfile: Dockerfile
restart: on-failure
ports:
- "9003:9003"
env_file:
- ./order/.env
networks:
- backend
volumes:
- ./order:/app/order
- /app/order/node_modules
logging:
driver: fluentd
options:
fluentd-async: "true"
fluentd-address: localhost:24224
tag: order-service
Here, in the docker-compose configuration, the used driver is fluentd as fluent-bit is also a part of fluentd system. in the options fluentd-async: true
will make an async connection with fluent-bit instance of the docker env. so that the order service will start the server in docker without any errors. the tag will be used in Grafana UI to identify each service.
To start our microservice application write down a command in the terminal docker-compose up
and it will start all docker containers together.
As per the docker-compose configuration file, the Grafana instance should be running on port 3000. open the Grafana instance in any browser but hit localhost URL: http://localhost:3000
. use the default username admin
and password admin
to get access to the portal. it will also ask you to enter your preferred password after login.
After login, it will show the dashboard screen.
Click on add a data source.
Click on Loki and enter URL: http://loki:3100
as we're using loki in docker. so we can use the docker container name to access the service in localhost.
Click on save & test
, it will save the data source in grafana admin. and then click on explore
to open the query dashboard.
Click on run query
to show logs for every job that is available in the grafana stack. if you're looking for specific logs of any service. you can query logs from the log browser
.
To learn more about the query language, you can check out the documentation provided by grafana on logQL.
Conclusion:
Adding centralized logging will make the microservice application easier to debug/monitor as we have single UI to monitor all services. usage of centralized logging will be more beneficial when the number of services will increase in the system and monitoring of services together gets difficult on the cloud platform. choosing a right logging stack will make a huge difference in debugging process of production application when it crashes.
Thanks for reading this. If you've any queries, feel free to email me at harsh.make1998@gmail.com.
Until next time!
Latest comments (0)