Running Kafka Locally: A Step-by-Step Guide
I am writing this because on the web there is no proper way to bootstrap kakfa locally and i spent few hours to figure out an issue !! frustrating right !
Introduction
Kafka is a powerful distributed streaming platform that is widely used for real-time data processing and messaging. To get started with Kafka without the need for a cloud-based environment, you can run it locally on your development machine. This guide will walk you through the process of setting up a local Kafka cluster using Docker Compose.
Prerequisites
- Docker installed on your system
Steps:
-
Create a Docker Compose File:
Create a file named
docker-compose.yml
with the following content:
version: '3'
services:
zookeeper:
image: wurstmeister/zookeeper:latest
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:2.11-1.1.1
ports:
- "9092:9092"
links:
- zookeeper
environment:
KAFKA_ADVERTISED_HOST_NAME: ${HOST_IP}
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
KAFKA_DELETE_TOPIC_ENABLE: 'true'
KAFKA_CREATE_TOPICS: "topic-test:1:1"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
KAFKA_ADVERTISED_HOST_NAME: ${HOST_IP}
This is very Important line while setting up kafka localy, this will allow local system to use IP and connect to kafka container, so get your IP quickly ! we can HOST_IP while booting up docker compose
- Start the Kafka Cluster: Run the following command in your terminal:
export HOST_IP=$(ifconfig | grep -E "([0-9]{1,3}\.){3}[0-9]{1,3}" | grep -v 127.0.0.1 | awk '{ print $2 }' | cut -f2 -d: | head -n1)
docker-compose up
This will start the Zookeeper and Kafka containers in the background.
Using Kafka
Once Kafka is running, you can produce and consume messages using a Kafka client library. Here's an example using the kafkajs
library:
const { Kafka } = require('kafkajs')
const ip = require('ip')
const host = process.env.HOST_IP || ip.address()
console.log(host)
const kafka = new Kafka({
clientId: 'my-app',
brokers: [`${host}:9092`],
})
const producer = kafka.producer()
const consumer = kafka.consumer({ groupId: 'test-group' })
const run = async () => {
// Producing
await producer.connect()
await producer.send({
topic: 'topic-test',
messages: [
{ value: 'Hello KafkaJS user!' },
],
})
// Consuming
await consumer.connect()
await consumer.subscribe({ topic: 'test-topic', fromBeginning: true })
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
console.log({
partition,
offset: message.offset,
value: message.value.toString(),
})
},
})
}
run().catch(console.error)
Additional Tips:
- Kafka Connect: Use Kafka Connect to integrate with other data sources and sinks.
- Schema Registry: Implement a schema registry to ensure data consistency and compatibility.
- Security: Configure security measures like authentication and authorization for your Kafka cluster.
By following these steps, you can set up a local Kafka cluster and start experimenting with its capabilities.
Top comments (0)