Apache Kafka: Docker Quick Start

Apache Kafka is a distributed streaming platform that can act as a message broker, as the heart of a stream processing pipeline, or even as the backbone of a large enterprise data synchronization system. Kafka is not only a highly-available and fault-tolerant system; it also handles vastly higher throughput compared to other message brokers such as RabbitMQ or ActiveMQ.

In this tutorial, you will utilize Docker & Docker Compose to run Apache Kafka & ZooKeeper. Docker with Docker Compose is the quickest way to get started with Apache Kafka and to experiment with clustering and the fault-tolerant properties Kafka provides. A full Docker Compose setup with 3 Kafka brokers and 1 ZooKeeper node can be found here.

Prerequisites

To complete this tutorial, you will need:

• A UNIX environment (Mac or Linux)
• Docker & Docker Compose

Note: Docker can be installed by following the official installation guide.

System Architecture

Before running Kafka with Docker, let's examine the architecture of a simple Apache Kafka setup.

• Kafka Cluster : A group of Kafka brokers forming a distributed system
• Kafka Broker : An instance of Kafka that holds topics of data
• ZooKeeper : A centralized system for storing and managing configuration
• Producer : A client that sends messages to a Kafka topic
• Consumer : A client that read messages from a Kafka topic

Kafka utilizes ZooKeeper to manage and coordinate brokers within a cluster. Producers and consumers are the main clients that interact with Kafka, which we'll take a look at once we have a running Kafka broker.

The above diagram shows the architecture of the systems we are going to run in this tutorial. It also helps demonstrate how Kafka brokers utilize ZooKeeper and shows the ports of the running services. In this tutorial, we'll start by running one Apache Kafka broker and one ZooKeeper node (seen above in blue). Later on, we'll form a three node cluster by adding in two more Kafka brokers (seen above in green).

Running ZooKeeper in Docker

Ensure you have Docker installed and running. You can verify this by running the following command; you should see a similar output.

docker -v
> Docker version 18.09.2, build 6247962

Additionally, verify you have Docker Compose installed:

docker-compose -v
> docker-compose version 1.23.2, build 1110ad01

We're ready to begin! Create a directory, such as ~/kafka, to store our Docker Compose files. Using your favorite text editor or IDE, create a file named docker-compose.yml in your new directory.

We'll start by getting ZooKeeper running. In the Docker Compose YAML file, define a zookeeper service as shown below:

version: '3'

services:
  zookeeper:
    image: zookeeper:3.4.9
    hostname: zookeeper
    ports:
      - "2181:2181"
    environment:
        ZOO_MY_ID: 1
        ZOO_PORT: 2181
        ZOO_SERVERS: server.1=zookeeper:2888:3888
    volumes:
      - ./data/zookeeper/data:/data
      - ./data/zookeeper/datalog:/datalog

A brief overview of what we're defining:

• Line 1: docker compose file version number, set to 3
• Line 4: starting the definition of a ZooKeeper service
• Line 5: The docker image to use for ZooKeeper and its version
• Line 6: The hostname the container will use when running
• Lines 7-8: The ports to expose to the host; ZooKeeper's default port
• Line 10: The unique ID of this ZooKeeper instance, set to 1
• Line 11: The port this ZooKeeper instance should run with
• Line 12: The list of ZooKeeper servers; in our case just one
• Lines 13-15: Mapping volumes on the host to store ZooKeeper data

Note: We've mapped ./data/zookeeper on the host to directories within the container. This allows ZooKeeper to persist data even if you destroy the container.

We can now start ZooKeeper by running the following command in the directory containing the docker-compose.yml file:

docker-compose up

Logs will start printing, and should end with a line similar to this:

zookeeper_1 | ... binding to port 0.0.0.0/0.0.0.0:2181

Congrats! ZooKeeper is running and exposed on port 2181. You can verify this utilizing netcat in a new terminal window:

echo ruok | nc localhost 2181
> imok

Running Kafka In Docker

We can now add our first kafka service to our Docker Compose file. We're calling this kafka2 as it will have a broker id of 2 and run on the default port of 9092. Later on, we'll add in kafka1 and kafka3. This is to demonstrate that order does not matter and broker ids are just for identification.

version: '3'

services:
...
  kafka2:
    image: confluentinc/cp-kafka:5.3.0
    hostname: kafka2
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka2:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_BROKER_ID: 2
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./data/kafka2/data:/var/lib/kafka/data
    depends_on:
      - zookeeper

If you prefer, copy the full gist found here. A brief overview of what we're defining:

• Line 6: The docker image to use for Kafka; we're using the Confluent image
• Line 7: The hostname this Kafka broker will use when running
• Line 8-9: The ports to expose; set to Kafka's default (9092)
• Line 11: Kafka's advertised listeners. Robin Moffatt has a great blog post about this.
• Line 12: Security protocols to use for each listener.
• Line 13: The inter-broker listener name (used for internal communication)
• Line 14: The list of ZooKeeper nodes Kafka should use
• Line 15: The broker ID of this Kafka broker.
• Line 16: The replication factor of the consumer offset topic (1 for one broker)
• Lines 17-18: Mapping volumes on the host to store Kafka data
• Lines 19-20: Start the ZooKeeper service before the Kafka service

Let's start the Kafka broker! In a new terminal window, run the following command in the same directory:

docker-compose up

ZooKeeper should still be running in another terminal, and if it isn't, Docker Compose will start it. You'll see a lot of logs being printed and then Kafka should be running! We can verify this by creating a topic.

If you have the Kafka command line tools installed, run:

kafka-topics --zookeeper localhost:2181 --create --topic new-topic --partitions 1 --replication-factor 1
> Created topic "new-topic".

If you don't have the Kafka command line tools installed, you can run a command using Docker as well:

docker exec -it kafka_kafka2_1 kafka-topics --zookeeper zookeeper:2181 --create --topic new-topic --partitions 1 --replication-factor 1
> Created topic "new-topic".

If you get any errors, verify both Kafka and ZooKeeper are running with docker ps and check the logs from the terminals running Docker Compose.

Yay! You now have the simplest Kafka cluster running within Docker. Kafka with broker id 2 is exposed on port 9092 and ZooKeeper on port 2181. Data for this Kafka cluster is stored in ./data/kafka2.

To stop the containers, you can use ctrl + c or cmd + c on the running Docker Compose terminal windows. If they don't stop, you can run docker-compose down. To remove the containers if they don't get removed as a part of down, you can run docker-compose rm.

Running Three Kafka Brokers In Docker

To run three brokers, we need to add two more kafka services to our Docker Compose file. We'll run broker 1 on port 9091 and broker 3 on port 9093.

Add two more services as so:

version: "3"

services:
...
  kafka1:
    image: confluentinc/cp-kafka:5.3.0
    hostname: kafka1
    ports:
      - "9091:9091"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19091,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9091
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./data/kafka1/data:/var/lib/kafka/data
    depends_on:
      - zookeeper

  kafka3:
    image: confluentinc/cp-kafka:5.3.0
    hostname: kafka3
    ports:
      - "9093:9093"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka3:19093,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_BROKER_ID: 3
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./data/kafka3/data:/var/lib/kafka/data
    depends_on:
      - zookeeper

You can find a full gist with ZooKeeper and three Kafka brokers here. Essentially, we update the ports, the broker ID, and the data directory on the host.

Note: In a production setup, you'd want the offset topic replication factor to be set higher than 1, but for the purposes of this tutorial I've left it at one since we started with one broker.

We can now verify that all three brokers are running by creating a topic with a replication factor of 3:

docker exec -it kafka_kafka2_1 kafka-topics --zookeeper zookeeper:2181 --create --topic three-isr --partitions 1 --replication-factor 3
> Created topic "three-isr".

If you receive an error, ensure all three Kafka clusters are running. Woohoo! You've now got a Kafka cluster with three brokers running.

Conclusion

Congrats! You've successfully started a local Kafka cluster using Docker and Docker Compose. Data is persisted outside of the container on the local machine which means you can delete containers and restart them without losing data. For next steps, I'd suggest playing around with Kafka's fault tolerance and replication features.

For example, you could create a topic with a replication factor of 3, produce some data, delete broker 2, delete broker 2's data directory (./data/kafka2), and start broker 2 and see that the data is replicated to the new broker. Pretty cool!

For full sets of Docker Compose files for running various Kafka Cluster setups, check out Stephane Maarek's kafka-stack-docker-compose repository. This post was inspired by it. :-).