No More ZooKeeper

Yes, you heard it right; there will be no more ZooKeeper with Kafka now but let me clarify you what I meant by that. As we are all aware of ZooKeeper has been integral part of Kafka, imagining Kafka cluster without ZooKeeper would have never been possible till now but things started changing since Kafka 2.8 release.

Kafka team have been working on project called KPI-500 to remove Kafka’s dependency on ZooKeeper. ZooKeeper is already in deprecated state since Kafka 3.5 release and in future Kafka4.0, ZooKeeper will be completely removed from Kafka, we will not need it to run Kafka Cluster.

Motivation Behind Removing ZooKeeper -
Till now ZooKeeper have played very pivotal role in any kafka cluster for metadata management. ZooKeeper cluster called Ensemble(as shown in below diagram) acts as a control center for Kafka, doing metadata management for leader selection, storing info about topic, partition, replica, consumer, offsets, etc. Over the period of time, we have relazed this ZooKeeper cluster comes with some drawbacks like-

• Single point of failure: If Zookeeper goes down, the entire Kafka cluster becomes unavailable.
• Scalability : Zookeeper metadata brings in limits on scalability of topics and partitions.
• High shutdown and recover time : ZooKeeper’s recovery time of several minutes for metadata recovery mechanism
• Resource Utilization: With ZooKeeper cluster, we need more memory and CPUs to run your Kafka cluster, leading to more cost.
• Increased complexity: Maintenance, support, administration, security of ZooKeeper cluster.

Kafka with ZooKeeper -

To address above limitation of Kafka with ZooKeeper, Kafka team have introduced a new way to run Kafka cluster without ZooKeeper i.e running Kafka cluster in Kraft mode. This replaces ZooKeeper with a self-managed metadata quorum, which is a set of Kafka brokers that manages metadata. The metadata quorum uses a consensus protocol called Kafka Raft (KRaft) to ensure that all brokers have the same view of metadata.

In Kafka with Zookeeper architecture, we have Zookeeper Quorum with a leader managing Kafka brokers and metadata. However, in a Quorum Controller setup (KRaft), only Kafka brokers are present, with one serving as the Quorum leader.
We can see, without requiring additional service like ZooKeeper, overall Kafka Cluster architecture becomes more simpler & easy to manage.

Kafka with Kraft-

How Leader Selection Works in Kraft Mode -
In KRaft mode, only the controller quorum (a subset of brokers) participates in the Raft protocol, while the rest of the brokers only communicate with the controller quorum. In KRaft mode, cluster metadata, reflecting the current state of all controller managed resources, is stored in a single partition Kafka topic called __cluster_metadata. KRaft uses this topic to synchronize cluster state changes across controller and broker nodes.

Brokers can be controllers only or brokers only, or both controller & broker. Lets see how leader selection, shown as Controller (Active) in above diagram works in Kraft mode -

• When a broker starts up, it sends a join request to the current leader to join the Raft cluster (kafka broker cluster). If there is no current leader, the broker will initiate an election and become the candidate to become a leader.
• The election process starts with a randomized timer, called the election timeout, that is started when the candidate becomes active.
• If the candidate does not receive a majority of votes from the other brokers within the election timeout, a new election is initiated with a longer timeout. This process continues until a candidate receives a majority of votes and becomes the leader.
• Once a leader is elected, it replicates the metadata to the other brokers in the cluster. If a broker detects that the leader has failed or become unresponsive, it initiates a new election to elect a new leader.

To run Kafka cluster with Kraft mode is fairly simple, we need to make few configuration changes in broker properties file. After configuring the brokers, you can start the them as usual. The brokers will automatically elect a leader and replicate metadata across the cluster using the Raft consensus protocol.