Send Logs from Go to Logstash in the ELK Stack with Docker Setup

Logging is a critical component of any application, allowing developers to easily identify and troubleshoot issues that arise during runtime. However, as applications become more complex, it can be challenging to manage and analyze logs in a way that is both efficient and effective. This is where Logstash comes in. Logstash is a powerful tool that simplifies the process of collecting, processing, and storing logs in a centralized location. In this blog post, we will explore how to push logs from a Go app to Logstash using the go-logstash package.

What is Logstash?

Logstash is an open-source tool that allows developers to easily ingest, process, and store logs. It is part of the Elastic Stack (also known as ELK), which includes Elasticsearch and Kibana. Logstash provides a variety of input and output plugins that allow it to collect logs from a wide range of sources, including files, TCP/UDP sockets, and messaging systems like Kafka and RabbitMQ. Once collected, Logstash can process and enrich the logs before forwarding them to Elasticsearch for storage and analysis.

Introducing go-logstash

go-logstash is a Golang package that provides a simple interface for pushing logs to Logstash. It supports both TCP and UDP protocols and can output logs in either JSON or string format. go-logstash is easy to use and provides customizable options for configuring the Logstash connection.

Setting up Logstash

Before we can start pushing logs to Logstash, we need to set up a Logstash instance. This can be done by downloading and installing Logstash from the Elastic website or setting it up using Docker Compose.

Create a new directory, let's call it go-logstash-demo and let's also create the necessary files that we need:

mkdir go-logstash-demo && cd go-logstash-demo
touch docker-compose.yml
touch docker-setup/logstash/Dockerfile
touch docker-setup/logstash/logstash.conf
touch docker-setup/go-logger/Dockerfile
touch main.go

Here is the docker-compose.yml file that creates the ELK stack, Elastic Search, Logstash, and Kibana each into its own container, along with a container for the Go application.

# docker-compose.yml

version: "3.9"

services:
  elasticsearch:
    image: elasticsearch:7.1.0
    volumes:
      - ./esdata:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - "discovery.type=single-node"
    networks:
      - elk

  logstash:
    build:
      context: .
      dockerfile: docker-setup/logstash/Dockerfile
    ports:
      - 9600:9600
      - 5228:5228
    environment:
      LOGSTASH_PORT: 5228
      LOGSTASH_INDEX: "test-index"
      ELASTIC_HOST: "elasticsearch:9200"
      ELASTIC_USERNAME: "elastic"
      ELASTIC_PASSWORD: "elastic"
    networks:
      - elk
    depends_on:
      - elasticsearch
    links:
      - elasticsearch

  kibana:
    image: kibana:7.1.0
    hostname: kibana
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch
    links:
      - elasticsearch
    environment:
      ELASTIC_HOST: "http://elasticsearch:9200"
      ELASTIC_USERNAME: "elastic"
      ELASTIC_PASSWORD: "elastic"

  go-app:
    container_name: go-app
    build:
      context: .
      dockerfile: docker-setup/go-logger/Dockerfile
    networks:
      - elk

networks:
  elk:
    driver: bridge

volumes:
  esdata:
    driver: local

Here is a custom Dockerfile for the Logstash container because we want to provide a custom logstash.conf file to it:

# docker-setup/logstash/Dockerfile

FROM docker.elastic.co/logstash/logstash-oss:7.1.0

COPY ./docker-setup/logstash/logstash.conf /etc/logstash/conf.d/

CMD logstash -f /etc/logstash/conf.d/logstash.conf

Here's an example of a basic logstash.conf file that creates a Logstash pipeline to collect logs from a TCP or UDP socket and output them to Elasticsearch:

# docker-setup/logstash/logstash.conf

input {
  tcp {
    host => "0.0.0.0"
    port => "${LOGSTASH_PORT}"
    codec => json_lines
  }
  udp {
    host => "0.0.0.0"
    port => "${LOGSTASH_PORT}"
    codec => json_lines
  }
}

output {
  stdout { codec => json_lines }
  elasticsearch {
      hosts => [ "${ELASTIC_HOST}" ]
      user => "${ELASTIC_USERNAME}"
      password => "${ELASTIC_PASSWORD}"
      codec => json_lines
      index => "${LOGSTASH_INDEX}"
  }
}

The logstash.conf config file is capable of supporting environment variables as well, which we are providing through our docker-compose.yml file. This pipeline listens for logs on TCP port 5228 and expects them to be in JSON format and outputs the logs to Elasticsearch in JSON.

We also need to create a Dockerfile for the Go application, as it would be using the internal Docker network to communicate with Logstash in order to push logs.

# docker-setup/go-logger/Dockerfile

FROM golang:1.19-alpine

WORKDIR /go-logstash-json

COPY . .

RUN go build -o out/logger *.go

CMD [ "./out/logger" ]

Pushing logs from a Go app to Logstash

Now that we have our ELK set up ready on our local, let's take a look at how we can push logs to it from a Go application using go-logstash.

Step 1: Create a Go application

In the root directory of go-logstash-demo, run:

go mod init example.com/go-logstash-demo

Step 2: Install go-logstash

The first step is to install go-logstash using the following command:

go get github.com/KaranJagtiani/go-logstash

Step 3: Test the Library

Next, we need to import the go-logstash package in our Go application and test the library:

# main.go

package main

import (
    logstash_logger "github.com/KaranJagtiani/go-logstash"
)

func main() {
    logger := logstash_logger.Init("logstash", 5228, "tcp", 5)

    payload := map[string]interface{}{
        "message": "TEST_MSG",
        "error":   false,
    }

    logger.Log(payload) // Generic log
    logger.Info(payload) // Adds "severity": "INFO"
    logger.Debug(payload) // Adds "severity": "DEBUG"
    logger.Warn(payload) // Adds "severity": "WARN"
    logger.Error(payload) // Adds "severity": "ERROR"
}

logger := logstash_logger.Init("logstash", 5228, "tcp", 5)

This line creates a new logger that will send logs to a Logstash instance running on logstash host in Docker through TCP port 5228 with a connection timeout of 5 seconds.

Conclusion

In conclusion, sending logs from a Go application to Logstash in the ELK stack is a powerful way to centralize and manage your application logs. With the help of the go-logstash library, it's easy to push logs to Logstash using either TCP or UDP. Logstash can then process and enrich the logs, making it easier to analyze and troubleshoot issues in your application.

Thank you for reading this beginner's guide on sending logs from a Go application to Logstash in the ELK stack with Docker setup. I hope that this guide has provided you with a useful overview of the process and has helped you get started with integrating Go and Logstash.

If you have any questions or feedback, please don't hesitate to leave a comment or reach out to me directly. I'm always happy to hear from readers and help in any way I can.

Thanks again for reading, and happy logging!