"Here we go, another article on setting up some thing in Docker 😒", you might think. This time it's Kafka. And I wouldn't write about it if it was that trivial.
We'll need two volumes for this guy. You know how it goes, go to "Volumes" side-menu and smash the "Add volume" button. Name the first one
zookeeper_data, and the second one -
Next go to the "Containers" side menu, click the "Add container" button. Set:
- name: zookeeper
- image: wurstmeister/zookeeper:latest
- publish port: 2181:2181
- add volumes:
- /opt/zookeeper-3.4.13/data to zookeeper_data
- /opt/zookeeper-3.4.13/conf to zookeeper_conf
Now destroy that "Deploy the container" button and wait a little.
This guy will need just one volume. Name it
kafka_volume. Now pay attention when creating a container. Set:
- name: kafka
- image: wurstmeister/kafka:latest
- publish port: 9092:9092
- add volumes:
- /kafka to kafka_data
- bind /var/run/docker.sock to /var/run/docker.sock
- KAFKA_ADVERTISED_HOST_NAME: 172.17.0.1
- KAFKA_ZOOKEEPER_CONNECT: 172.17.0.1:2181
- ZK: 172.17.0.1:2181
- HOST: 172.17.0.1:9092
- KAFKA_ADVERTISED_PORT: 9092
HOST envs are just for shortening commands later on.
There is already a guide on how to use this stuff, but there is a mistake in the consuming part.
But let's start with creating a topic. Go to ">_ Console" of the Kafka container. Or execute command below in your terminal session
docker exec -it kafka /bin/bash
Once inside container create topic just as in above mentioned guide:
kafka-topics.sh --create --topic topic \ --partitions 4 --zookeeper $ZK --replication-factor 1 kafka-topics.sh --describe --topic topic --zookeeper $ZK
You'll see messages about topic creation and it's description:
Created topic topic. Topic: topic PartitionCount: 4 ReplicationFactor: 1 Configs: Topic: topic Partition: 0 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: topic Partition: 1 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: topic Partition: 2 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: topic Partition: 3 Leader: 1002 Replicas: 1002 Isr: 1002
The guide says to run this:
kafka-console-consumer.sh --topic=topic --zookeeper=$ZK
But if you try to do that you'll get an error about
zookeeper is not a recognized option.
So the correct command is this:
kafka-console-consumer.sh --topic=topic --bootstrap-server=$HOST
It didn't freeze, it just waits for messages to come.
Open another console or terminal session and connect to the kafka container. Execute:
kafka-console-producer.sh --topic=topic \ --broker-list=`broker-list.sh`
Now you can type some messages here and it'll show in another session where we have a consumer 🚀
Inside kafka_app in the project repo, you'll find two short files with producer and consumer. This is just to show how we can use Kafka in Python:
python -m venv venv source venv/bin/activate pip install -r requirements.txt python consumer.py
In another terminal session:
source venv/bin/activate python producer.py
Now look at the terminal with our python consumer and you'll see a topic, message key and value.
Also, if you didn't close previous sessions with the console consumer, you'll see the same message there. And publishing from the console producer will deliver a message to both console and python consumers.
Now you have one more tool in your toolbox! Go use this stuff 💪