Apache Kafka without Zookeeper and Dedicated Controllers

Are you interested in setting up Kafka without Zookeeper and with a dedicated controller quorum? Here are the steps and reference project showcasing how to do this using the Confluent community-licensed container images. A Grafana dashboard to observe the new metrics is also provided.

Introduction

Kafka Raft brings the consensus protocol into the Controller Plane of Apache Kafka from Zookeeper. With this change, the role of a Kafka instance can be that of a controller, broker, or both. Getting this configuration stood up requires some tweaks to the confluent cp-kafka image.

If you want a deeper understanding of the design and implementation details, check out Jun Rao's course on Kafka Internals. Specifically, the control-plane section.

Configuration

There are many configuration parameters with Apache Kafka; highlighted here are the ones necessary to build out the cluster with KRaft.

• node.id

The property node.id replaces broker.id. Be sure that all identifiers in the cluster are unique across brokers and controllers.

• process.roles

A node can be both a broker or a controller. Set to broker,controller to enable both the controller and data planes on a node.

• controller.listener.names

List out the listener names used for the controller. This indicates to a node the listener to use for that communication. While the property is a list, just like advertised.listeners, the first one is what is used for controller communication.

• controller.quorum.voters

A comma delimited list of voters in the control plane, where a controller is noted as: node_id@hostname:port.

example:

KAFKA_CONTROLLER_QUORUM_VOTERS: 10@controller-0:9093,11@controller-1:9093,12@controller-2:9093

Additional Configuration

There are other tuning parameters for the controller plan, see the documentation for details.

Lesson Learned

Do not remove cluster settings from the dedicated controllers, since a controller is the node that performs administration operations, such as creating a topic.

Incorrectly removing these from the controllers caused topics to be created without Apache Kafka defaults.

KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_NUM_PARTITIONS: 4

Storage

Another change to setting up Apache Kafka with Raft is the storage. The storage on each node must be configured, before starting the JVM. This can be done with a kafka-storage command provided as part of Apache Kafka.

• Generate a unique UUID for the cluster, you can use kafka-storage random-uuid or another means.

• Before starting the cluster, format the metadata storage with kafka-storage format.
  

kafka-storage format -t $KAFKA_CLUSTER_ID -c <server.properties>

Container Images

With these details in mind, applying them to Confluent's cp-kafka image takes a little finesse, at least with version 7.3.0. The cp-kafka container's entry point, /etc/confluent/docker/run, builds the configuration for Apache Kafka from environment variables following conventions. In addition, there are validation steps to catch misconfiguration. These validations, however, need to change, since certain assumptions no longer apply in a zookeeper-less setup. In addition, a node that is only for a controller does not define advertised.listeners so validation for that needs to be removed.

Script Modifications

The following tasks need to be done to start up these images with raft.

• Remove KAFKA_ZOOKEEPER_CONNECT validation for all nodes.
• Remove checking for zookeeper ready state for all nodes.
• Remove KAFKA_ADVERTISED_LISTENERS validation for dedicated controller nodes.
• Create Metadata store for all nodes by running kafka-storage format.

Command

The cp-kafka image's command is /etc/confluent/docker/run, and the scripts and docker-compose command setting, allow
these containers to start with raft consensus protocol.

• broker.sh

volumes:
  - ./broker.sh:/tmp/broker.sh
command: bash -c '/tmp/broker.sh && /etc/confluent/docker/run'

• controller.sh

volumes:
  - ./controller.sh:/tmp/controller.sh
command: bash -c '/tmp/controller.sh && /etc/confluent/docker/run'

Seeing It In Action

If you are interested in seeing all this in action, check out the kafka-raft docker-compose
setup in the dev-local project. It is a fully working examples with 3 controllers and 4 brokers.

Metrics

If you are going to deploy Kafka with Raft to production, having visibility to metrics is important. Adding that visibility is just as important (if not more so) than having dedicated controllers. The key to dashboards, ensure that they report correctly on data and control planes are separated or combined.

Grafana

A Grafana Dashboard is a multi-step process of extracting the metrics and storing them in a time-series database (e.g. Prometheus) and then visualizing those collected metrics in a Grafana Dashboard.

JMX Prometheus Exporter

The KRaft Monitor metrics are defined in the documentation, with an MBean name, such as kafka.server:type=raft-metrics,name=current-state. Using a JMX client, such as jmxterm, shows the MBean is name is just kafka.server:type=raft-metrics with each metric an attribute in that bean. This is different from the current documentation.

If you deploy Java applications with JMX Metrics in containers, I highly recommend jmxterm.

java -jar jmxterm.jar

In the cp containers, the java process is process 1, but use the command jvms to see all available processes; and verify the JVM's process-id is indeed 1.

$>open 1

Show all the MBeans in the JVM, with beans.

$>beans
...
kafka.server:type=raft-metrics
...

Select a bean and use info to explore the attributes on a bean and get to get the current value of an attribute.

$>bean kafka.server:type=raft-metrics
#bean is set to kafka.server:type=raft-metrics

$>info
# attributes
%0   - append-records-rate (double, r)
%1   - commit-latency-avg (double, r)
%2   - commit-latency-max (double, r)
%3   - current-epoch (double, r)
%4   - current-leader (double, r)
%5   - current-state (double, r)
%6   - current-vote (double, r)
%7   - election-latency-avg (double, r)
%8   - election-latency-max (double, r)
%9   - fetch-records-rate (double, r)
%10  - high-watermark (double, r)
%11  - log-end-epoch (double, r)
%12  - log-end-offset (double, r)
%13  - number-unknown-voter-connections (double, r)
%14  - poll-idle-ratio-avg (double, r)

$>get current-leader
current-leader = 10.0;

Leveraging the above, the following properly exposes these from JMX Prometheus Exporter. Since current-state attribute value is a string, its value needs to be associated with a label to capture it in Prometheus.

rules:
- pattern: "kafka.server<type=raft-metrics><>(current-state): (.+)"
  name: kafka_server_raft_metrics
  labels:
  name: $1
  state: $2
  value: 1
- pattern: "kafka.server<type=raft-metrics><>(.+): (.+)"
  name: kafka_server_raft_metrics
  labels:
  name: $1

Grafana Dashboard

Grafana is great for custom configuration, but that means time and effort are needed to build them. Here is a dashboard around some of those raft metrics; it is not a complete dashboard.

Node Information

So with metrics emitting, put them into a Grafana dashboard.

By using the current-state we can see who is the leader, in addition to capturing node.id. In addition, the dashboard component joins in other data to provide additional information on each node.

Node Counts

Counts of nodes and active controller are always re-assuring, and this leverages an existing metric, kafka_controller_kafkacontroller_value{name="ActiveControllerCount",}. This metric only is emitted from a controller, so by counting the existence of this metric you see the number of controllers in the cluster, and by summing the value of the metric you get the actual number of active controllers; alert if this is ever not equal to 1.

Check out the dashboard to see how the other values are calculated, as it is the same as in the zookeeper-based installations.

Active Controller

To get the node.id of the quorum leader, just find max(kafka_server_raft_metrics{name="current-leader",}). Because scraping of each node is from slightly different times, different values are possible at the time of a change; max is used to make the single value display easy to build.

If a new leader is being voted, that will show up in the voted leader metric. In a single value dashboard, I do not expect this to be that useful, but in a time-series history of values, more value would be in having this metric recorded.

Full Dashboard

Here is an example dashboard that also captures the fetch rate of the controller metadata.

Open-Source Tools

I have checked a variety of open-source tools out there and have been unsuccessful in seeing raft metrics. Active controller information is incorrectly displayed. None of the tools tried came to support raft, but it is important that before you upgrade, you have a proper monitoring and alerting strategy in place.

Summary

Be sure you properly test your monitoring and Apache Kafka support infrastructure as part of your move to Kafka Raft Consensus Protocol. Also, validate that kafka-raft metrics are captured and ensure that dashboards and tools work when brokers and controllers are on separate nodes.