3 Simple steps to set up Kafka locally using Docker

In this article, let’s go over how you can set up Kafka and have it running on your local environment. For this, make sure that you have Docker installed on your machine.

If you want to get a brief understanding of Kafka and how it works, I’d recommend you go through this article, as it’d help you get a good understanding of Kafka and some basic kafka terminology.

Now, Let’s get started with setting up Kafka locally using Docker

1. Download and Install Kafka:

With Docker installed, you can follow the below steps in order to download the spotify/kafka image on your machine and run the image as a docker container

• Download spotify/kafka image using docker

docker pull spotify/kafka

• Create the docker container using the downloaded image

docker run -p 2181:2181 -p 9092:9092 --name kafka-docker-container --env ADVERTISED_HOST=127.0.0.1 --env ADVERTISED_PORT=9092 spotify/kafka

What the above steps do is — a. Download the spotify/kafka image and b. Using this image, create a docker container named kafka-docker-container (you can name the container anything you prefer) with ports 2181, 9092 of your machine mapped to 2181, 9092 of the container.

One nice thing about the spotify/kafka docker image is that it comes with both Kafka and Zookeeper configured in the same image, so you don’t have to worry about having to configure and start Kafka and Zookeeper separately.

With that, you have Kafka running on your machine and open to listening and storing messages from producers which can then be consumed by the consumers.

2. Create your very first Kafka topic:

Since, there are currently no topics present on the Kafka broker, you can go ahead and create one to see if everything works as expected

To use the Kafka CLI, you have to:

• Open your terminal and exec inside the kafka container

docker exec -it kafka-docker-container bash

• Once in the container, goto the below path

cd /opt/kafka_2.11-0.10.1.0/

Here 2.11 is the Scala version and 0.10.1.0 is the Kafka version that is used by the spotify/kafka docker image

• Create a topic

bin/kafka-topics.sh —create —topic my-first-kafka-topic —zookeeper localhost:2181 —partitions 1 —replication-factor 1

The above command would have created your first topic with the name my-first-kafka-topic in your kafka container.

For simplicity, I have set the partitions and the replication factor as 1 for the topic, but you can always play around with this configuration. If you don’t know what partitions and replication factor mean in the Kafka context, I’d recommend you to go through this article in order to get a good understanding.

3. List all the Kafka topics:

After you’ve created the topic as mentioned above, you can run the below command in order to list all the topics present on your locally running Kafka container:

bin/kafka-topics.sh —list —zookeeper localhost:2181

And if everything goes well, you should be able to see the topic you just created being listed after you run the above command.

More Kafka articles that you can go through:

1. Apache Kafka: A Basic Intro
2. Kafka Partitions and Consumer Groups in 6 mins

Follow for the next Kafka blog in the series. I shall also be posting more articles talking about Software engineering concepts.