As of Kafka 7.5 Zookeeper is deprecated as an option to start new kafka cluster and default strategy is to use the KRaft to have a zookeeper less kafka.
As of now there is not much available online over how to setup this new style of Kafka cluster without zookeeper. Even ChatGPT doesn't know if you were to ask it, see below
Most of the examples that you will find online are from 7.0 to 7.2 which required you to run some scripts to not only generate a cluster id before the cluster starts but also run a pre hook script to disable or remove zookeeper options from the configs before the cluster would come online and not check for zookeeper.
Many examples use bitnami/kafka image to show how to setup a kraft based examples but I found them to be inconsistent in their setup and use non standard flags.
ChatGPT use wurstmeister/kafka-docker in its example so I decided to check it but its based on zookeeper. Nonetheless it is quite good to know what you need to setup a kafka cluster as the settings are still the same and only zookeeper is no more in the kraft mode.
For Kafka-UI I used provectus/kafka-ui which has a lot of good examples over how to setup kafka-ui with a kraft based kafka cluster but they are all pre 7.5.x where you required lot of tinkering before you can have the cluster up and running.
To setup monitoring kafka cluster I refered to this amazing repo streamthoughts/kafka-monitoring-stack-docker-compose that has detailed examples over how to use jmx-exporter to have the metrics available for prometheus scrapping and highly recommend to refer to it over other information available which are not in depth
One thing to take note of is for volume mapping Kafka requires that you create a folder and its owner is set to 1000:1000 before you can crate the mapping for your container.
Below is the config to setup a 2 node (leaving the 3rd node as an exercise) cluster with monitoring and UI to monitor your cluster. Not adding grafana and prometheus to the post as that is out of the scope. To try out you can use Grafana cloud before setting it up on your own
#KAFAK 10.0.3.52
kafka_00:
image: "confluentinc/cp-kafka:7.5.1"
hostname: "kafak_00"
user: "0:0"
container_name: kakashi_kafka_00
networks:
network:
ipv4_address: 10.0.3.52
aliases:
- kakashi_kafka_00
expose:
- "9092"
- "9093"
- "9094"
- "9101"
volumes:
- "/opt/kafka-00-data:/var/lib/kafka/data"
environment:
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
- KAFKA_BROKER_ID=0
- KAFKA_ADVERTISED_LISTENERS=INTERNAL://kafka_00:9092,EXTERNAL://kafka_00:9094
- KAFKA_CONTROLLER_QUORUM_VOTERS=0@kafka_00:9093,1@kafka_01:9093
- KAFKA_DEFAULT_REPLICATION_FACTOR=2
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
- KAFKA_LISTENERS=CONTROLLER://kafka_00:9093,EXTERNAL://kafka_00:9094,INTERNAL://kafka_00:9092
- KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
- KAFKA_NODE_ID=0
- KAFKA_NUM_PARTITIONS=2
- KAFKA_PROCESS_ROLES=broker,controller
- KAFKA_ENABLE_KRAFT=yes
- CLUSTER_ID=45vUBEgyQjSywv_811biQw
- KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
- KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0
- KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1
- KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
- KAFKA_JMX_PORT=9101
- KAFKA_JMX_HOSTNAME=kafka_00
- KAFAK_JMX_OPTS=-Djava.rmi.server.hostname=kafka_00 -Djava.rmi.server.port=9101 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
cpu_shares: 64
mem_reservation: 1024m
mem_limit: 1512m
restart: unless-stopped
#KAFAK_01 10.0.3.54
kafka_01:
image: "confluentinc/cp-kafka:7.5.1"
hostname: "kafka_01"
user: "0:0"
container_name: kakashi_kafka_01
networks:
network:
ipv4_address: 10.0.3.54
aliases:
- kakashi_kafka_01
expose:
- "9092"
- "9093"
- "9094"
- "9101"
volumes:
- "/opt/kafka-01-data:/var/lib/kafka/data"
environment:
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
- KAFKA_BROKER_ID=1
- KAFKA_ADVERTISED_LISTENERS=INTERNAL://kafka_01:9092,EXTERNAL://kafka_01:9094
- KAFKA_CONTROLLER_QUORUM_VOTERS=0@kafka_00:9093,1@kafka_01:9093
- KAFKA_DEFAULT_REPLICATION_FACTOR=2
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
- KAFKA_LISTENERS=CONTROLLER://kafka_01:9093,EXTERNAL://kafka_01:9094,INTERNAL://kafka_01:9092
- KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
- KAFKA_NODE_ID=1
- KAFKA_NUM_PARTITIONS=2
- KAFKA_PROCESS_ROLES=broker,controller
- KAFKA_ENABLE_KRAFT=yes
- CLUSTER_ID=45vUBEgyQjSywv_811biQw
- KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
- KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0
- KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1
- KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
- KAFKA_JMX_PORT=9101
- KAFKA_JMX_HOSTNAME=kafka_01
- KAFAK_JMX_OPTS=-Djava.rmi.server.hostname=kafka_01 -Djava.rmi.server.port=9101 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
cpu_shares: 64
mem_reservation: 1024m
mem_limit: 1512m
restart: unless-stopped
#KAFAK_UI 10.0.3.56
kafka_ui:
image: "provectuslabs/kafka-ui:53a6553765a806eda9905c43bfcfe09da6812035"
container_name: kakashi_kafka_ui
networks:
network:
ipv4_address: 10.0.3.56
aliases:
- kakashi_kafka_ui
depends_on:
- kafka_00
- kafka_01
expose:
- "8080"
environment:
- AUTH_TYPE=LOGIN_FORM
- DYNAMIC_CONFIG_ENABLED=true
- SPRING_SECURITY_USER_NAME=${KAFKA_UI_USERNAME}
- SPRING_SECURITY_USER_PASSWORD=${KAFKA_UI_PASSWORD}
- SERVER_SERVLET_CONTEXT_PATH=/ka5ka-u1
- KAFKA_CLUSTERS_0_NAME=kakashi-kafka-00
- KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka_00:9094
- KAFKA_CLUSTERS_0_METRICS_PORT=9101
- KAFKA_CLUSTERS_1_NAME=kakashi-kafka-01
- KAFKA_CLUSTERS_1_BOOTSTRAPSERVERS=kafka_01:9094
- KAFKA_CLUSTERS_1_METRICS_PORT=9101
cpu_shares: 64
mem_reservation: 512m
mem_limit: 786m
restart: unless-stopped
#KAFKA_00_JMX 10.0.3.58
kafka_00_jmx:
image: "sscaling/jmx-prometheus-exporter:0.12.0"
container_name: kakashi_kafka_00_jmx
networks:
network:
ipv4_address: 10.0.3.58
aliases:
- kakashi_kafka_00_jmx
depends_on:
- kafka_00
expose:
- "5556"
environment:
- CONFIG_YML=/etc/jmx_exporter/config.yml
- JVM_OPTS=-Xmx128M
volumes:
- ./kafka_jmx_exporter/config/jmx_exporter/config_kafka_00.yml:/etc/jmx_exporter/config.yml
cpu_shares: 64
mem_reservation: 196m
mem_limit: 256m
restart: unless-stopped
#KAFKA_01_JMX 10.0.3.60
kafka_01_jmx:
image: "sscaling/jmx-prometheus-exporter:0.12.0"
container_name: kakashi_kafka_01_jmx
networks:
network:
ipv4_address: 10.0.3.60
aliases:
- kakashi_kafka_01_jmx
depends_on:
- kafka_01
expose:
- "5556"
environment:
- CONFIG_YML=/etc/jmx_exporter/config.yml
- JVM_OPTS=-Xmx128M
volumes:
- ./kafka_jmx_exporter/config/jmx_exporter/config_kafka_01.yml:/etc/jmx_exporter/config.yml
cpu_shares: 64
mem_reservation: 196m
mem_limit: 256m
restart: unless-stopped
Kafka without zookeeper is surely a big step forward and easier to manage.
Top comments (0)