DEV Community

donvitocodes
donvitocodes

Posted on • Originally published at melvinvivas.com on

Ingest NGINX container access logs to ElasticSearch using Fluentd and Docker

Ingest NGINX container access logs to ElasticSearch using Fluentd and Docker

This is an example on how to ingest NGINX container access logs to ElasticSearch using Fluentd and Docker. I also added Kibana for easy viewing of the access logs saved in ElasticSearch.

Ingest NGINX container access logs to ElasticSearch using Fluentd and Docker

I noticed that ElasticSearch and Kibana needs more memory to start faster so I've increased my docker engine's memory configuration to use 6GB. I am using version 2.0.0.0-mac78 (28905) of Docker CE in a Mac.

Ingest NGINX container access logs to ElasticSearch using Fluentd and Docker

I've prepared the compose file "docker-compose.yml" for this example. All files are also available in my github repo. The compose file below starts 4 docker containers ElasticSearch, Fluentd, Kibana and NGINX. I wasn't able to find a Fluentd docker image which has the ElasticSearch plugin built-in so I just created a new docker image and uploaded it to my dockerhub repo. The Dockerfile for the custom fluentd docker image can also be found in my github repo.

GitHub logo donvito / docker-elasticsearch-fluentd-nginx

This is an example on how to run ingest Nginx access logs to ElasticSearch using Fluentd. Also added Kibana for easy viewing of the access logs saved in ElasticSearch.

docker-elasticsearch-fluentd-nginx

This is an example on how to run ingest Nginx access logs to ElasticSearch using Fluentd. Also added Kibana for easy viewing of the access logs saved in ElasticSearch.

Please check out my blog post about this example https://www.melvinvivas.com/docker-elasticsearch-fluentd-nginx/




The compose file also creates a volume for ElasticSearch so you don't lose data when you restart the ElasticSearch container. It also creates a new network named "logging-net" where all the containers will be communicating.

For the example to work, we need to configure Fluentd as the logging driver of our NGINX container.

logging:
      driver: fluentd  
      options:
        fluentd-address: localhost:24224
        tag: httpd.access

docker-compose.yml

version: "3.3"
services:

  elasticsearch:
    image: 'docker.elastic.co/elasticsearch/elasticsearch:6.5.1'
    ports:
      - "9200:9200"
      - "9300:9300"
    volumes:
      - elasticsearch-vol:/usr/share/elasticsearch/data 
    networks:
      - logging-net
    environment:
      - xpack.security.enabled=false 

  fluentd:
    image: melvindave/fluentd-elastic:1.0
    ports:
      - "24224:24224"
    networks:
      - logging-net
    depends_on:  
      - "elasticsearch"
    volumes:
      - ./fluentd/etc:/fluentd/etc      

  kibana:
    image: 'docker.elastic.co/kibana/kibana:6.5.1'
    ports:
      - "5601:5601"
    networks:
      - logging-net
    depends_on:
      - elasticsearch    

  nginx:
    image: nginx
    ports:
      - "80:80"
    networks:
      - logging-net
    deploy:
      replicas: 1  
    volumes:
      - ${NGINX_HTML_DIR}:/usr/share/nginx/html
    logging:
      driver: fluentd  
      options:
        fluentd-address: localhost:24224
        tag: httpd.access
    depends_on:  
      - "fluentd" 

networks:
  logging-net:

volumes:
  elasticsearch-vol:

To run the example, you need to set an environment variable to point to the root directory where you want NGINX to serve your html files from. If you want to use the sample index.html in the repo, just execute this command.

$ export NGINX_HTML_DIR=./nginx-files/

Once you've got the environment variable configured, you can now run all the containers using this command.

$ docker-compose up -d

You'll see a similar output like below.

Ingest NGINX container access logs to ElasticSearch using Fluentd and Docker

Wait a few minutes since ElasticSearch and Kibana takes some time to startup. You can constantly check if ElasticSearch is up by accessing it from the browser.

Ingest NGINX container access logs to ElasticSearch using Fluentd and Docker

Once Kibana is up, you need to create an index pattern to be able to see the NGINX access logs. In the fluentd configuration we used fluentd as the prefix by configuring logstash_prefix.

<source>
  @type forward
  port 24224
</source>

<filter **>
  @type stdout
</filter>

<match *.**>
    @type copy
    <store>
      @type elasticsearch
      host elasticsearch
      port 9200
      logstash_format true
      logstash_prefix fluentd
      logstash_dateformat %Y%m%d
      include_tag_key true
      type_name access_log
      tag_key @log_name
      flush_interval 1s
    </store>
</match>

So let's proceed to create the index pattern in Kibana. Use fluentd* to capture all indices prefixed with fluentd.

Ingest NGINX container access logs to ElasticSearch using Fluentd and Docker
`

Ingest NGINX container access logs to ElasticSearch using Fluentd and Docker

Ingest NGINX container access logs to ElasticSearch using Fluentd and Docker

Initially, we won't see any NGINX access logs in Kibana because we haven't really accessed Nginx. To access NGINX, just go to http://localhost from your browser. We've configured NGINX to run in its default port 80 so the port in the URL is not required. Once we've accessed the default page we created, we would start seeing access logs in Kibana.

Ingest NGINX container access logs to ElasticSearch using Fluentd and Docker

Here's how the log entries would look like in Kibana.

Ingest NGINX container access logs to ElasticSearch using Fluentd and Docker

That's it! Hope you find it useful!

For more updates on new blog posts and sample codes, you can follow me in Twitter @donvito and GitHub. Most of my tweets are about Docker, Kubernetes, and Golang. I also share code in my GitHub. If you want to know more about what I do, please add me in LinkedIn. I recently started a new youtube channel - I upload some tutorials there as well. Check it out!

Oldest comments (2)

Collapse
 
dimastatz profile image
Dima Statz • Edited

Hi Melvin,
thanks a lot for your article.

I am doing pretty similar task - ingesting data to AWS S3 and Apache Kafka by using Nginx and Fluentd. Here you can find more info about it github.com/dimastatz/no-code-data-..., medium.com/swlh/no-code-data-colle....

The questions is - did you tested scenario when Elasticsearch is going down for couple of minutes and then goes up again. Is it possible to prevent data loss in Nginx&Fluentd ?
Thanks,
Dima

Collapse
 
donvitocodes profile image
donvitocodes

Hi Dima, glad you liked the article. The code I did was for a coding test and is not production-ready. So no, I haven't tested it when ElasticSearch is going down.