DEV Community

shun
shun

Posted on • Edited on

To set up Logstash and Elasticsearch in separate containers and send log data between them

#logstash #elasticsearch

Create a Docker network named "elastic" to connect both containers: 

docker network create elastic
Enter fullscreen mode Exit fullscreen mode

Connect the prepared Logstash container to the "elastic" network: 

docker network connect elastic logstash
Enter fullscreen mode Exit fullscreen mode

Create and run the Elasticsearch Docker container, connecting it to the "elastic" network: 

docker run --network=elastic -p 9200:9200 -p 9300:9300 --name=elasticsearch -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.15.2
Enter fullscreen mode Exit fullscreen mode

After running Elasticsearch, stop it once with Ctrl+C.

Reconnect Elasticsearch to the "elastic" network: 

docker network connect elastic elasticsearch
Enter fullscreen mode Exit fullscreen mode

Start the Elasticsearch container: 

docker start elasticsearch
Enter fullscreen mode Exit fullscreen mode

Find the IP address of the Elasticsearch container (Note: In Windows, use double quotes for the command): 

docker inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}" elasticsearch
Enter fullscreen mode Exit fullscreen mode

The command docker network inspect elastic is used to retrieve information about the Docker network named "elastic". This command displays detailed information about the network, including its name, ID, creation timestamp, and configuration. 

docker network inspect elastic
Enter fullscreen mode Exit fullscreen mode

Output:

"Containers": {
  "<CONTAINER_ID>": {
    "Name": "elasticsearch",
    "EndpointID": "<EndpointID>",
    "MacAddress": "00:00:ac:00:00:00",
    "IPv4Address": "172.19.0.3/16",
    "IPv6Address": ""
  },
  "<CONTAINER_ID>": {
    "Name": "logstash",
    "EndpointID": "<EndpointID>",
    "MacAddress": "00:00:ac:00:00:00",
    "IPv4Address": "172.19.0.2/16",
    "IPv6Address": ""
  }
},
Enter fullscreen mode Exit fullscreen mode

Start Logstash container: 

docker start logstash
Enter fullscreen mode Exit fullscreen mode

Access the Logstash container: 

docker exec -it logstash /bin/bash
Enter fullscreen mode Exit fullscreen mode

Update the Logstash pipeline configuration: 

vi /usr/share/logstash/pipeline/logstash.conf
Enter fullscreen mode Exit fullscreen mode

Enter the following configuration in the editor: 

input {
  file {
    path => "/usr/share/logstash/logs/test.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}

output {
  elasticsearch {
    hosts => ["http://172.19.0.3:9200"]
    index => "test_index"
  }
  stdout { codec => rubydebug }
}
Enter fullscreen mode Exit fullscreen mode

Save and exit the editor.

Execute Logstash with the updated pipeline configuration:

/usr/share/logstash/bin/logstash -f /usr/share/logstash/pipeline/logstash.conf
Enter fullscreen mode Exit fullscreen mode

The Logstash process will start, and it will read the data from the specified input file and send it to Elasticsearch, as well as display the output using stdout with the rubydebug codec.

You should see the following output:

{
        "host" => "000000000000",
  "@timestamp" => "2023-00-00T00:00:00.000Z",
        "test2" => "test2",
        "test3" => "test3",
        "test" => "test1",
    "@version" => "1",
        "path" => "/usr/share/logstash/logs/test.log"
}
Enter fullscreen mode Exit fullscreen mode

Now, let's access the Elasticsearch container:

Access the Elasticsearch container:

docker exec -it elasticsearch /bin/bash
Enter fullscreen mode Exit fullscreen mode

To check the count of documents in the "test_index" index: 

curl 172.19.0.3:9200/test_index/_count
Enter fullscreen mode Exit fullscreen mode

Output:

{"count":1,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
Enter fullscreen mode Exit fullscreen mode

To retrieve and pretty-print the log data from the "test_index" index: 

curl 172.19.0.3:9200/test_index/_search?pretty
Enter fullscreen mode Exit fullscreen mode

Output:

{
  "_index" : "test_index",
  "_type" : "_doc",
  "_id" : "AAAAAAAAAAaaaaaaaaaa",
  "_score" : 1.0,
  "_source" : {
    "host" : "000000000000",
    "@timestamp" : "2023-00-00T00:00:00.000Z",
    "test2" : "test2",
    "test3" : "test3",
    "test" : "test1",
    "@version" : "1",
    "path" : "/usr/share/logstash/logs/test.log"
  }
}
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

quanphat profile image

How to integrate Elasticsearch in Express

Le Quan Phat -

ravikishan profile image

Building a Web Search Engine in Go with Elasticsearch

Ravi Kishan -

ugifractal profile image

github action services: mysql, redis and elasticsearch

sugiarto -

absterellio profile image

Using an Elasticsearch client with Typescript and React to search for data

absterellio -