Getting Start with Kafka using Docker

In this article, we will be using Kafka via docker and try producing and consuming messages. We will learn some keywords and basic tools in Kafka.

Firstly, let's create a new folder named kafka (or whatever name of your project), and go to that folder

mkdir kafka && cd kafka

Inside the kafka folder, we create a new docker-compose.yml file with the content below:

version: '3.8'

services:
  kafka: # service name, can change to any name
    image: 'bitnami/kafka:3.6.2' # docker image from https://hub.docker.com/r/bitnami/kafka
    environment:
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER

Run Kafka server

docker-compose up -d

We need the container name to access into it. Check the NAMES column when running:

docker ps
---
CONTAINER ID   IMAGE                      COMMAND                  CREATED        STATUS          PORTS           NAMES
f9258b058804   bitnami/kafka:3.6.2        "/opt/bitnami/script…"   24 hours ago   Up 39 minutes   9092/tcp        kafka_kafka_1

Access into docker container command line (In my case in kafka_kafka_1, please change to your service name if mismatch)

docker exec -it kafka_kafka_1 bash

Now you are in the container and can use a set of tools below:

• kafka-topics.sh: Manage topics
• kafka-console-producer.sh: Manage producer
• kafka-console-consumer.sh: Manage consumer

Kafka treat message as "event". Events and grouped into a category called "topic".
Assume we are creating a chat application. Let's create a new topic called "chat-message" which is a stream of message records, as well as create a new topic called "user-presence" to manage the online status of users. We will use the server kafka:9092 mentioned in docker-compose configuration

kafka-topics.sh --create --topic chat-message --bootstrap-server kafka:9092
kafka-topics.sh --create --topic user-presence --bootstrap-server kafka:9092

We can check how many topics we have by using:

kafka-topics.sh --list --bootstrap-server kafka:9092
---
__consumer_offsets
chat-message
user-presence

And view detail each topic:

kafka-topics.sh --describe --topic chat-message --bootstrap-server kafka:9092
---
Topic: chat-message TopicId: 0XIXOlZ-TAWqEBlBUP7WAA PartitionCount: 1   ReplicationFactor: 1    Configs:
    Topic: chat-message Partition: 0    Leader: 0   Replicas: 0 Isr: 0

Now we can use 2 separate terminals to set up the producer and consumer, and see how the consumer get message from the producer in real time.

In the first terminal, let's use the tool kafka-console-consumer.sh to setup a consumer to waiting for the messages in the topic chat-message to consume:

kafka-console-consumer.sh --topic chat-message --from-beginning --bootstrap-server kafka:9092

Open the second terminal and use the tool kafka-console-producer.sh to setup the producer by running the command:

kafka-console-producer.sh --topic chat-message --bootstrap-server kafka:9092

The kafka-console-producer.sh will waiting for our inputs. Let's try pasting 3 events below and hit enter:

Event: {"message_id": 1, "sender_id": "user1", "receiver_id": "user2", "content": "Hello, how are you?", "timestamp": "2024-04-10T09:00:00"}
Event: {"message_id": 2, "sender_id": "user2", "receiver_id": "user1", "content": "I'm good, thanks! How about you?", "timestamp": "2024-04-10T09:01:00"}
Event3: some random words...

Note that the message format is plaintext, in the example above we use both plain text and some JSON-like format to show that the format is not strict. The consumer will need to have logic to decode the plaintext to some meaningful messages.

Now we should see events appear in the consumer terminal

That is the very basic setup of Kafka using docker. If you have any questions please ask in the comment section.

Bonus:
If you don't want to access the container, in the host machine you can replace the bash with the kafka tools like kafka-topics.sh to run the Kafka command outside of container like this:

docker exec -it kafka_kafka_1 kafka-topics.sh --list --bootstrap-server kafka:9092

Original port: https://www.anvo.dev/articles/getting-start-with-kafka-using-docker