DEV Community

loading...

Installing Apache Kafka on Windows (For Development)

Dendi Handian
I am that backend programmer who doesn't return 200 OK for errors and doesn't write only POST method for all endpoints.
Updated on ・5 min read

Prerequisites

Have Java 8 or greater installed on your machine. Open the command prompt and execute this to confirm it is installed:

java -version
Enter fullscreen mode Exit fullscreen mode

Meanwhile I'm using Java 11

Download The Apache Kafka Binary

Go to https://kafka.apache.org/downloads, if you're using Scala on your machine then find the matched version, if not download any latest available version Kafka binary.

kafka-download-page

Let's say I got kafka_2.12-2.8.0.tgz. From here on, I will use my home directory C:\Users\dendi so you can use yours to follow along. I extracted it into C:\Users\dendi and renamed the folder from kafka_2.12-2.8.0 into kafka. So the complete path is C:\Users\dendi\kafka.

Here is the structure inside the kafka folder:

kafka
    |__ bin
    |__ config
    |__ libs
    |__ licenses
    |__ logs
    |__ site-docs
    |__ LICENSE (file)
    |__ NOTICE (file)

Enter fullscreen mode Exit fullscreen mode

Provide Data Folder For Zookeeper and Kafka

In the kafka directory, let's create another folder named data and create two folders named zookeeper and kafka inside the data folder.

kafka
    |__ bin
    |__ config
    |__ data
        |__ kafka
        |__ zookeeper
    |__ libs
    |__ licenses
    |__ logs
    |__ site-docs
    |__ LICENSE (file)
    |__ NOTICE (file)

Enter fullscreen mode Exit fullscreen mode

Configure Zookeeper and Kafka Server

Find the config\zookeeper.properties file, open it with code editor and find the dataDir variable and fill it with:

...

dataDir=data/zookeeper

...
Enter fullscreen mode Exit fullscreen mode

And then find the config\server.properties file, open it with code editor and find the log.dirs variable and fill it with:

...

log.dirs=data/kafka

...
Enter fullscreen mode Exit fullscreen mode

If you're using app like Terminus or Hyper, it's very useful for managing consoles because we will open 4-5 consoles later.

Running Zookeeper

Now let's open our 1st command prompt, go to kafka directory, and execute this command:

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Enter fullscreen mode Exit fullscreen mode

if the result similar like this, then it's already running:

[2021-06-06 11:09:01,739] INFO Server environment:java.io.tmpdir=C:\Users\dendi\AppData\Local\Temp\ (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,739] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,741] INFO Server environment:os.name=Windows 10 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,742] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,743] INFO Server environment:os.version=10.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,743] INFO Server environment:user.name=dendi (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,743] INFO Server environment:user.home=C:\Users\dendi (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,744] INFO Server environment:user.dir=C:\Users\dendi\kafka (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,744] INFO Server environment:os.memory.free=492MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,745] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,745] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,747] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,748] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,755] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir data\zookeeper\version-2 snapdir data\zookeeper\version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-06-06 11:09:01,802] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-06-06 11:09:01,806] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-06-06 11:09:01,810] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-06-06 11:09:01,834] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2021-06-06 11:09:01,841] INFO Reading snapshot data\zookeeper\version-2\snapshot.0 (org.apache.zookeeper.server.persistence.FileSnap)
[2021-06-06 11:09:01,891] INFO Snapshotting: 0x8d to data\zookeeper\version-2\snapshot.8d (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-06-06 11:09:01,919] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2021-06-06 11:09:01,931] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)
Enter fullscreen mode Exit fullscreen mode

Running Kafka

Let's open our 2nd command prompt, go to kafka directory and execute this command:

.\bin\windows\kafka-server-start.bat .\config\server.properties
Enter fullscreen mode Exit fullscreen mode

Then your kafka server should be running, hopefully there is no error log in console result. But we will test if it actually running well or not by creating a topic in next step.

Creating Topic

Let's open our 3rd command prompt, go to kafka directory, we will create a simple topic named test with this command:

.\bin\windows\kafka-topics.bat --create --topic test --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

We could tell if it was successfuly created when we see this result:

Created topic test.
Enter fullscreen mode Exit fullscreen mode

Running Consumer

We could use the same 3rd command prompt after we creating topic, now we will use the built-in kafka consumer. We already inside kafka folder, then execute this command to start the consumer for our created test topic:

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
Enter fullscreen mode Exit fullscreen mode

After executing it, you will see absolutely nothing because the consumer is running and streaming the test topic and there is no data available at the moment. Don't close this command prompt, we will see if it's actually working if we produce a message.

Running Producer

Let's open our 4th command prompt, go to kafka directory, and let's run the built-in producer for the test topic by this command:

.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
Enter fullscreen mode Exit fullscreen mode

It will open interactive session that we can type anything in it and hit enter to produce the message to the topic. Try type anything and hit enter, see the typed message is showed in the consumer (3rd command prompt). If you see it, then the kafka is running perfectly and you're ready to use it for your project development.


version used:

java version "11.0.10" 2021-01-19 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.10+8-LTS-162)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.10+8-LTS-162, mixed mode)

kafka 2.12-2.8.0

Enter fullscreen mode Exit fullscreen mode

Discussion (1)

Collapse
gklijs profile image
Gerard Klijs

For development I much rather use a docker setup. Easy to destroy and recreate. Or the Confluent cli, which is great when you need multiple components of the Confluent platform.