Getting Started on Kafka (Windows)
1. Start up zookeeper server
Open a command prompt in C:\kafka and enter this command:
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Zookeeper is running on Port 2181
2. Start the Kafka server
Open a new command prompt and start the Kafka server:
.\bin\windows\kafka-server-start.bat .\config\server.properties
3. Create a topic
Open a new command prompt in C:\kafka\bin\windows:
kafka-topics.bat --create --bootstrap-server localhost:9092 --topic mytopic
4. Create a Producer Console
kafka-console-producer.sh --broker-list <BROKER_LIST> --topic <TOPIC_NAME>
Open another new command prompt in C:\kafka\bin\windows:
kafka-console-producer.bat --broker-list localhost:9092 --topic mytopic
You should see a > prompt when the producer is ready.
5. Create a Consumer Console
kafka-console-consumer.sh --bootstrap-server <BROKER_LIST> --topic <TOPIC_NAME>
Open another new command prompt in C:\kafka\bin\windows:
kafka-console-consumer.bat --topic mytopic --bootstrap-server localhost:9092 --from-beginning
6. Paste Data to Producer and Observe Consumer
Sample data for use:
{"Name":"Kenneth","Age":"35","Gender":"Male"}
{"Name":"Gemma","Age":"25","Gender":"Female"}
{"Name":"Amy","Age":"15","Gender":"Male"}
Referencing the image below, once you paste a row of data into the producer terminal (left), the data appears on the consumer terminal (right) almost instantaneously.
In Git Bash/ Linux
Similar commands collated from kafka quickstart
# Download and extract
tar -xzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0
# Start the ZooKeeper service
bin/zookeeper-server-start.sh config/zookeeper.properties
# Start and hide logs on terminal
bin/zookeeper-server-start.sh config/zookeeper.properties > /dev/null 2>&1 &
# verify that Zookeeper is indeed running
ps -aef
# Start the Kafka broker service in another terminal
bin/kafka-server-start.sh config/server.properties
# Create a topic in another terminal
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
# Show details of a topic
bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
Topic: quickstart-events TopicId: z_zqwwkiT6ueXVJxi3EAdA PartitionCount: 1 ReplicationFactor: 1 Configs:
Topic: quickstart-events Partition: 0 Leader: 0 Replicas: 0 Isr: 0
# Run Producer Console in another terminal
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
# Run Consumer Console in another terminal
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
# Write event to topic at Producer console
This is my first message
This is my second message
Demonstrating 3 Brokers Replica
# Create 2 other brokers
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties
ll config/serv*
Update these 3 areas in server-1 and server-2 file:
# SERVER-1
# id of the broker
broker.id=1
# broker port
listeners=PLAINTEXT://:9093
# log files directories
log.dirs=/tmp/kafka-logs-1
# SERVER-2
# id of the broker
broker.id=2
# broker port
listeners=PLAINTEXT://:9094
# log files directories
log.dirs=/tmp/kafka-logs-2
Now start all 3 brokers
bin/kafka-server-start.sh config/server.properties
bin/kafka-server-start.sh config/server-1.properties
bin/kafka-server-start.sh config/server-2.properties
Now Create a topic with 2 partitions and 3 replication factor across 3 brokers
bin/kafka-topics.sh --create --topic TestTopicXYZ --partitions 2 --replication-factor 3 --bootstrap-server localhost:9092, localhost:9093, localhost:9094
Lastly describe the topic to examine:
bin/kafka-topics.sh --describe --topic TestTopicXYZ --bootstrap-server localhost:9092,localhost:9093, localhost:9094
Topic: TestTopicXYZ TopicId: 65thxHOeTIGSJJ5L_jl-XQ PartitionCount: 2 ReplicationFactor: 3 Configs:
Topic: TestTopicXYZ Partition: 0 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
Topic: TestTopicXYZ Partition: 1 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
What is Kafka?
- Publish/Subscribe systems
With Publish/Subscribe systems, each event can be processed by multiple consumers who are listening to the topic. So multiple consumers can all get the same messages from the producer.
Pub/Sub systems introduce the concept of a topic. We can have multiple topics (to categorize your messages) and multiple producers writing to each topic and multiple consumers reading from a topic.
Distributed
Kafka is like a messaging system. Kafka Cluster is made up of more than 1 Kafka servers. Each Kafka server is referred to as a Broker. A Kafka cluster will have multiple brokers, therefore it is distributed application.Fault-Tolerant
In Kafka cluster, messages are replicated in multiple brokers. A message published in Broker 1 will also be published in Broker 3. Therefore Kafka is fault-tolerant as message is replicated and not lost even if Broker 1 is down.Language Agnostic
Data transferred (called Messages) are byte arrays so you can use JSON or a data format like Avro. Messages are immutable and have a timestamp, a value and optional key/headers
Kafka Architecture
Every Kafka ecosystem can have multiple producers/publishers.
A Kafka system consists of several components:
Producer - Any service that pushes messages to Kafka is a producer.
Consumer - Any service that consumes (pulls) messages from Kafka is a consumer. There can be multiple Consumer groups in the Kafka ecosystem. Consumers must be associated in a Consumer groups.
Kafka Brokers - the kafka cluster is composed of a network of nodes called brokers. Each machine/instance/container running a Kafka process is called a broker.
In a Kafka cluster, the individual servers are referred to as brokers. Brokers will handle:
- Receiving Messages - accepting messages from producers
- Storing Messages - storing them on disk organized by topic. Each message has a unique offset identifying it.
- Serving Messages - sending messages to consumer services when the consumer requests it
- Topics - A Kafka server has multiple topics. Every topics can have multiple partitions. Messages in Kafka are stored in topics, which are the core structure for organizing and storing your messages. You can think of a topic as a log file that stores all the messages and maintains their order. New message are appended to the end of the topic.
Offset Explorer
A UI tool for Kafka to help us visualize kafka clusters, topics and offset etc.
Thank you!
Top comments (0)