DEV Community

shun
shun

Posted on • Updated on

To set up Logstash and Elasticsearch in separate containers and send log data between them

#logstash #elasticsearch

Create a Docker network named "elastic" to connect both containers: 

docker network create elastic
Enter fullscreen mode Exit fullscreen mode

Connect the prepared Logstash container to the "elastic" network: 

docker network connect elastic logstash
Enter fullscreen mode Exit fullscreen mode

Create and run the Elasticsearch Docker container, connecting it to the "elastic" network: 

docker run --network=elastic -p 9200:9200 -p 9300:9300 --name=elasticsearch -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.15.2
Enter fullscreen mode Exit fullscreen mode

After running Elasticsearch, stop it once with Ctrl+C.

Reconnect Elasticsearch to the "elastic" network: 

docker network connect elastic elasticsearch
Enter fullscreen mode Exit fullscreen mode

Start the Elasticsearch container: 

docker start elasticsearch
Enter fullscreen mode Exit fullscreen mode

Find the IP address of the Elasticsearch container (Note: In Windows, use double quotes for the command): 

docker inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}" elasticsearch
Enter fullscreen mode Exit fullscreen mode

The command docker network inspect elastic is used to retrieve information about the Docker network named "elastic". This command displays detailed information about the network, including its name, ID, creation timestamp, and configuration. 

docker network inspect elastic
Enter fullscreen mode Exit fullscreen mode

Output:

"Containers": {
  "<CONTAINER_ID>": {
    "Name": "elasticsearch",
    "EndpointID": "<EndpointID>",
    "MacAddress": "00:00:ac:00:00:00",
    "IPv4Address": "172.19.0.3/16",
    "IPv6Address": ""
  },
  "<CONTAINER_ID>": {
    "Name": "logstash",
    "EndpointID": "<EndpointID>",
    "MacAddress": "00:00:ac:00:00:00",
    "IPv4Address": "172.19.0.2/16",
    "IPv6Address": ""
  }
},
Enter fullscreen mode Exit fullscreen mode

Start Logstash container: 

docker start logstash
Enter fullscreen mode Exit fullscreen mode

Access the Logstash container: 

docker exec -it logstash /bin/bash
Enter fullscreen mode Exit fullscreen mode

Update the Logstash pipeline configuration: 

vi /usr/share/logstash/pipeline/logstash.conf
Enter fullscreen mode Exit fullscreen mode

Enter the following configuration in the editor: 

input {
  file {
    path => "/usr/share/logstash/logs/test.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}

output {
  elasticsearch {
    hosts => ["http://172.19.0.3:9200"]
    index => "test_index"
  }
  stdout { codec => rubydebug }
}
Enter fullscreen mode Exit fullscreen mode

Save and exit the editor.

Execute Logstash with the updated pipeline configuration:

/usr/share/logstash/bin/logstash -f /usr/share/logstash/pipeline/logstash.conf
Enter fullscreen mode Exit fullscreen mode

The Logstash process will start, and it will read the data from the specified input file and send it to Elasticsearch, as well as display the output using stdout with the rubydebug codec.

You should see the following output:

{
        "host" => "000000000000",
  "@timestamp" => "2023-00-00T00:00:00.000Z",
        "test2" => "test2",
        "test3" => "test3",
        "test" => "test1",
    "@version" => "1",
        "path" => "/usr/share/logstash/logs/test.log"
}
Enter fullscreen mode Exit fullscreen mode

Now, let's access the Elasticsearch container:

Access the Elasticsearch container:

docker exec -it elasticsearch /bin/bash
Enter fullscreen mode Exit fullscreen mode

To check the count of documents in the "test_index" index: 

curl 172.19.0.3:9200/test_index/_count
Enter fullscreen mode Exit fullscreen mode

Output:

{"count":1,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
Enter fullscreen mode Exit fullscreen mode

To retrieve and pretty-print the log data from the "test_index" index: 

curl 172.19.0.3:9200/test_index/_search?pretty
Enter fullscreen mode Exit fullscreen mode

Output:

{
  "_index" : "test_index",
  "_type" : "_doc",
  "_id" : "AAAAAAAAAAaaaaaaaaaa",
  "_score" : 1.0,
  "_source" : {
    "host" : "000000000000",
    "@timestamp" : "2023-00-00T00:00:00.000Z",
    "test2" : "test2",
    "test3" : "test3",
    "test" : "test1",
    "@version" : "1",
    "path" : "/usr/share/logstash/logs/test.log"
  }
}
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

fazly_fathhy profile image

Top 5 Containerization Tools You Should Know in 2024 for DevOps Success

Fazly Fathhy -

kresi profile image

🌟 Introducing the My 3D Model Viewer! 🌟

Alberts Kresi -

koolkamalkishor profile image

Evaluating Medical Retrieval-Augmented Generation (RAG) with NVIDIA AI Endpoints and Ragas

KAMAL KISHOR -

tharunikal profile image

Tricentis Tosca: A Powerful Tool for Continuous Testing

Tharunika L -