This is an example on how to ingest NGINX container access logs to ElasticSearch using Fluentd and Docker. I also added Kibana for easy viewing of the access logs saved in ElasticSearch.
I noticed that ElasticSearch and Kibana needs more memory to start faster so I've increased my docker engine's memory configuration to use 6GB. I am using version 2.0.0.0-mac78 (28905) of Docker CE in a Mac.
I've prepared the compose file "docker-compose.yml" for this example. All files are also available in my github repo. The compose file below starts 4 docker containers ElasticSearch, Fluentd, Kibana and NGINX. I wasn't able to find a Fluentd docker image which has the ElasticSearch plugin built-in so I just created a new docker image and uploaded it to my dockerhub repo. The Dockerfile for the custom fluentd docker image can also be found in my github repo.
donvito / docker-elasticsearch-fluentd-nginx
This is an example on how to run ingest Nginx access logs to ElasticSearch using Fluentd. Also added Kibana for easy viewing of the access logs saved in ElasticSearch.
docker-elasticsearch-fluentd-nginx
This is an example on how to run ingest Nginx access logs to ElasticSearch using Fluentd. Also added Kibana for easy viewing of the access logs saved in ElasticSearch.
Please check out my blog post about this example https://www.melvinvivas.com/docker-elasticsearch-fluentd-nginx/
The compose file also creates a volume for ElasticSearch so you don't lose data when you restart the ElasticSearch container. It also creates a new network named "logging-net" where all the containers will be communicating.
For the example to work, we need to configure Fluentd as the logging driver of our NGINX container.
logging:
driver: fluentd
options:
fluentd-address: localhost:24224
tag: httpd.access
docker-compose.yml
version: "3.3"
services:
elasticsearch:
image: 'docker.elastic.co/elasticsearch/elasticsearch:6.5.1'
ports:
- "9200:9200"
- "9300:9300"
volumes:
- elasticsearch-vol:/usr/share/elasticsearch/data
networks:
- logging-net
environment:
- xpack.security.enabled=false
fluentd:
image: melvindave/fluentd-elastic:1.0
ports:
- "24224:24224"
networks:
- logging-net
depends_on:
- "elasticsearch"
volumes:
- ./fluentd/etc:/fluentd/etc
kibana:
image: 'docker.elastic.co/kibana/kibana:6.5.1'
ports:
- "5601:5601"
networks:
- logging-net
depends_on:
- elasticsearch
nginx:
image: nginx
ports:
- "80:80"
networks:
- logging-net
deploy:
replicas: 1
volumes:
- ${NGINX_HTML_DIR}:/usr/share/nginx/html
logging:
driver: fluentd
options:
fluentd-address: localhost:24224
tag: httpd.access
depends_on:
- "fluentd"
networks:
logging-net:
volumes:
elasticsearch-vol:
To run the example, you need to set an environment variable to point to the root directory where you want NGINX to serve your html files from. If you want to use the sample index.html in the repo, just execute this command.
$ export NGINX_HTML_DIR=./nginx-files/
Once you've got the environment variable configured, you can now run all the containers using this command.
$ docker-compose up -d
You'll see a similar output like below.
Wait a few minutes since ElasticSearch and Kibana takes some time to startup. You can constantly check if ElasticSearch is up by accessing it from the browser.
Once Kibana is up, you need to create an index pattern to be able to see the NGINX access logs. In the fluentd configuration we used fluentd as the prefix by configuring logstash_prefix.
<source>
@type forward
port 24224
</source>
<filter **>
@type stdout
</filter>
<match *.**>
@type copy
<store>
@type elasticsearch
host elasticsearch
port 9200
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 1s
</store>
</match>
So let's proceed to create the index pattern in Kibana. Use fluentd* to capture all indices prefixed with fluentd.
Initially, we won't see any NGINX access logs in Kibana because we haven't really accessed Nginx. To access NGINX, just go to http://localhost from your browser. We've configured NGINX to run in its default port 80 so the port in the URL is not required. Once we've accessed the default page we created, we would start seeing access logs in Kibana.
Here's how the log entries would look like in Kibana.
That's it! Hope you find it useful!
For more updates on new blog posts and sample codes, you can follow me in Twitter @donvito and GitHub. Most of my tweets are about Docker, Kubernetes, and Golang. I also share code in my GitHub. If you want to know more about what I do, please add me in LinkedIn. I recently started a new youtube channel - I upload some tutorials there as well. Check it out!
Top comments (2)
Hi Melvin,
thanks a lot for your article.
I am doing pretty similar task - ingesting data to AWS S3 and Apache Kafka by using Nginx and Fluentd. Here you can find more info about it github.com/dimastatz/no-code-data-..., medium.com/swlh/no-code-data-colle....
The questions is - did you tested scenario when Elasticsearch is going down for couple of minutes and then goes up again. Is it possible to prevent data loss in Nginx&Fluentd ?
Thanks,
Dima
Hi Dima, glad you liked the article. The code I did was for a coding test and is not production-ready. So no, I haven't tested it when ElasticSearch is going down.