Unlock Real-Time Data Streaming in 10 Steps

Mastering the fundamentals of Apache Kafka is crucial for effective real-time data streaming. In this article, we will explore the essential commands for operating Kafka Broker, producing and consuming messages, and understanding topic and offset details. For a more in-depth look at Apache Kafka setup and CLI mastery, refer to our step-by-step guide at https://t8tech.com/it/data/unlock-real-time-data-streaming-a-step-by-step-guide-to-apache-kafka-setup-and-cli-mastery/.

This self-contained setup provides a comprehensive overview of the basic setup and functionality using the command-line interface.

Let's dive into the key commands:

1. Begin by downloading the latest version of Kafka, currently version 2.3.0, from Apache Kafka.

2. Extract the downloaded artifact using the command. This will yield a folder named kafka_2.11-2.3.0.

tar xvf kafka_2.11-2.3.0.tgz

3. Navigate to the kafka_2.11-2.3.0/bin directory.

4. Initialize the Zookeeper server, a crucial step before launching Kafka Broker.

./zookeeper-server-start.sh ../config/zookeeper.properties

5. With the Zookeeper server initialized, start Kafka Broker using the following command:

./kafka-server-start.sh ../config/server.properties

6. Create a topic called ‘csptest’ with two partitions.

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic csptest

7. Next, start two listeners on topic csptest. The same command can be used in two separate terminals, allowing consumption from both partitions. Note that the group is set to topic_group for both listeners.

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic csptest --group topic_group

8. Now, start a producer/publisher with the following command. Then, produce 5 messages.

./kafka-console-producer.sh --broker-list localhost:9092 --topic csptest

>entry-1

>entry-2

>entry-3

>entry-4

>entry-5

9. Upon inspecting the terminals of both listeners, we notice that the consumed messages are distributed in a cyclical manner.

$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic csptest --group topic_group

entry-2

entry-4



$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic csptest --group topic_group

entry-1

entry-3

entry-5

10. Now, let’s retrieve the topic details, including the partition count, leader, and replicas. This information is particularly valuable in a clustered environment.

$ ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic csptest

Topic:csptest
PartitionCount:2
ReplicationFactor:1
Configs:

Topic: csptest
Partition: 0
Leader: 0
Replicas: 0
Isr: 0

Topic: csptest
Partition: 1
Leader: 0
Replicas: 0
Isr: 0

11. We can obtain consumer details and offset information for each partition using the following command:

$ ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group topic_group --describe



GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID

topic_group csptest 0 3 3 0 consumer-1-379adec4-08e7-4a13-8e26-91c4fe10a3a8 /127.0.0.1 consumer-1

topic_group csptest 1 2 2 0 consumer-1-85381523-5103-4bd0-a523-4ca09f41a6a7 /127.0.0.1 consumer-1

12. We can list all topics using the following command:

$ ./kafka-topics.sh --list --zookeeper localhost:2181

__consumer_offsets

csptest

my-topic

13. The topic _consumer_offsets, which is the default and already available in the Kafka Broker store, stores offset information in the Broker. We can browse this topic using the following command:

$ ./kafka-console-consumer.sh --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --bootstrap-server localhost:9092 --topic __consumer_offsets

[topic_group,csptest,1]::OffsetAndMetadata(offset=2, leaderEpoch=Optional[0], metadata="", timestamp=1566047971652, expirationTimestamp=None)

[topic_group,csptest,0]::OffsetAndMetadata(offset=3, leaderEpoch=Optional[0], metadata="", timestamp=1566047971655, expirationTimestamp=None)

I trust this demonstration has shed valuable light on the subject and provided you with a wealth of insight.