DEV Community

loading...
Cover image for Dockerized Elastic Stack

Dockerized Elastic Stack

thehoodsdev profile image Kiya Abdulahi Updated on ・4 min read

Introduction

Hello, my name is Kiya Abdulahi and I'm new to the Dev community! I'm beyond excited to learn from all of you and contribute back to Dev!

Background

I come from a technology operations background. I've worked as an Incident Response Manager at Target for a little over two years. In that role, monitoring the health of applications/infrastructure and being able to respond to sudden degradations or outages was extremely critical. It's only right that I have strong feelings about how observable applications are.

In order to build observable applications, we can leverage technology like the Elastic Stack to help us observe our applications. Let's take a look at how we can set up the Elastic Stack.

Observability

Although observability sounds very much like a new tech buzzword, it has been around since the early 1930s.

In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. — Wikipedia

If our applications are not observable, no amount of eyes on glass monitoring will be sufficient. Far too often we make the mistake of building dashboards and alerting after a major outage. As the architecture of our applications become more complex, this approach no longer scales. We have to build observable applications from the jump!

Okay, I'll get off my soapbox now and get to the good stuff!

Prerequisites

  • Some knowledge of Docker
  • Docker installed
    • Install Docker on a Mac here
    • Install Docker on Windows here
  • 4.0 GB of memory allocated to docker
    • Docker Preferences > Resources > Advanced

Elastic Stack

The Elastic Stack was developed by Elastic and is arguably the most popular, Open-Source, log management platform today. It consists of Elasticsearch, Logstash and Kibana.

  • Elasticsearch is a RESTful search and analytics engine which centrally stores your data

  • Kibana lets you visualize your Elasticsearch data

  • Logstash lets us consume data from multiple sources, transform that data, then ship it to our favorite "stash". We won't cover Logstash for now but it will be part of the series!

Running Elastic Stack Locally

Let's start by opening up our favorite terminal (mine is iTerm2 with ohmyzsh configured) and create a few directories and files by running:

mkdir elastic-stack && cd elastic-stack && touch docker-compose.yml && mkdir elasticsearch kibana && cd elasticsearch && touch elasticsearch.yml && cd ../kibana && touch kibana.yml && cd ..

Enter fullscreen mode Exit fullscreen mode

We just:

  • Made our root directory called elastic-stack
  • Created a docker-compose.yml file inside elastic-stack
  • Created two more directories inside elastic-stack called elasticsearch and kibana
  • Added a yml file in the elasticsearch and kibana directories

Now let's open up our elastic-stack folder with our favorite code editor or IDE (I'm using Visual Studio Code for this example) and start configuring our Elastic Stack.

Inside our elasticsearch.yml file, paste in the following:

cluster.name: es-docker-cluster # setting the name of our docker cluster
network.host: 0.0.0.0 # making elasticsearch accessible to all IPs on the host machine
Enter fullscreen mode Exit fullscreen mode

Inside our kibana.yml file, paste in the following:

server.name: kibana # setting the name of our Kibana server
server.host: 0.0.0.0 # making kibana accessible to all IPs on the host machine
elasticsearch.hosts: [ "http://elasticsearch:9200" ] # allows communication from Kibana to Elasticsearch.
Enter fullscreen mode Exit fullscreen mode

Inside our docker-compose.yml file, paste in the following:

version: '3.7' # file version is specific to Docker Engine release versions. learn more here: https://docs.docker.com/compose/compose-file/compose-versioning/
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0 # latest elasticsearch docker image from elastic
    container_name: elasticsearch # setting our elasticsearch container name
    environment:
      node.name: es01 # setting our node name inside the 
      cluster.initial_master_nodes: es01 # declaring our es01 node as master
      ES_JAVA_OPTS: "-Xms512m -Xmx512m" # setting our JVM heap size. learn more here: https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
    volumes:
      - ./elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml # volumes allow us to map our elasticsearch.yml file from the host machine to our docker container. learn more about volumes here: https://docs.docker.com/storage/volumes/
    ports:
      - 9200:9200 # setting elasticsearch accessibility on port 9200 on our host machine and docker container. these are the default ports for elasticsearch
    networks:
      - elastic  # referencing a docker network. more details below under networks

  kibana:
    image: docker.elastic.co/kibana/kibana:7.6.0 # latest elasticsearch docker image from elastic
    container_name: kibana # setting our kibana container name
    volumes:
      - ./kibana/kibana.yml:/usr/share/kibana/config/kibana.yml # volumes allow us to map our kibana.yml file from the host machine to our docker container. learn more about volumes here: https://docs.docker.com/storage/volumes/
    ports:
      - 5601:5601 # setting kibana accessibility on port 5601 on our host machine and docker container. these are the default ports for kibana
    networks:
      - elastic # referencing a docker network. more details below under networks

networks:
  elastic:
    driver: bridge # default network driver. learn more here: https://docs.docker.com/network/bridge/
Enter fullscreen mode Exit fullscreen mode

Now let's run the following command to build and start our docker containers!

docker-compose up
Enter fullscreen mode Exit fullscreen mode

Test if Elasticsearch and Kibana are running

You can do this via your favorite browser.

  • Go to localhost:9200 and you should see:
{
  "name" : "es01",
  "cluster_name" : "es-docker-cluster",
  "cluster_uuid" : "Ce7adjQNQ-qz2eIRKHIL9Q",
  "version" : {
    "number" : "7.6.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "7f634e9f44834fbc12724506cc1da681b0c3b1e3",
    "build_date" : "2020-02-06T00:09:00.449973Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
Enter fullscreen mode Exit fullscreen mode
  • Go to localhost:5601 and you should see:

Alt Text

Nice Job

You did it! You now have a dockerized instance of Elastic Stack running locally on your machine!

Alt Text

You can find a copy of the dockerized Elastic Stack configuration you just put together here.

Up next

This is the first part of an Elastic Stack series I'm writing. Be sure to stay tuned for the following:

  • Securing our dockerized Elastic Stack
  • Shipping logs to our dockerized Elastic stack with Logstash
  • Querying and visualizing our logs
  • Alerting based off of our logs
  • Deploying our dockerized Elastic Stack to production

Discussion (0)

pic
Editor guide