DEV Community

Cover image for How to integrate kafka with nodejs ?
Chafroud Tarek
Chafroud Tarek

Posted on • Updated on

How to integrate kafka with nodejs ?

Kafka is a powerful, open-source stream processing platform that has become increasingly popular in recent years for its ability to handle large amounts of data and real-time streaming scenarios. Node.js, on the other hand, is a JavaScript runtime built on Chrome's V8 JavaScript engine, known for its ability to create scalable, high-performance network applications. In this blog post, we'll explore the ways in which Kafka and Node.js can be used together to build efficient, real-time data processing systems that can handle high volumes of data and traffic.

👋 You can also check my weekly posts in Linkedin

Requirements:

  • Basic knowledge of nodejs
  • Basic Knowledge of docker
  • Basic knowledge of javascript

What we will do:

  1. Set up kafka.
  2. Set up two Node.js apps: one for the producer and one for the consumer.
  3. sent message from producer and consume it
  4. scenario: A user will be automatically created in the consumer when a producer is created.

Discuss some crucial terms:

  • producer: a producer is a program that writes data to a Kafka cluster. The producer sends messages to one or more topics.
  • consumer: a consumer is a program that reads data from a Kafka cluster.
  • borker: A broker is a Kafka server that stores and serves data.
  • topic: topics are the channels through which producers and consumers communicate
  • partition: a partition is a unit of data storage in a topic. Each topic is divided into one or more partitions

Let's play the game:

To begin, we will install Kafka

  • Install Docker: Kafka is typically run in a Docker container, so you will need to install Docker on your system if it is not already installed. Follow the instructions for your operating system to install Docker.

  • we will need two Docker images:

    • wurstmeister/zookeeper
    • wurstmeister/kafka

1️⃣ Create a docker-compose.yml file and include the following in it.

version: "3"
services:
zookeeper:
image: 'wurstmeister/zookeeper:latest'
ports:
- '2181:2181'
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: 'wurstmeister/kafka:latest'
ports:
- '9092:9092'
environment:
- KAFKA_BROKER_ID=1
- KAFKA_LISTENERS=PLAINTEXT://:9092
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper

2️⃣ Run the following command to start the containers (-d to start the containers in the foreground)

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

3️⃣ Start kafka shell:(change kafka_kafka_1 by your container name)

docker exec -it <u>kafka_kafka_1</u> /bin/sh
Enter fullscreen mode Exit fullscreen mode

4️⃣ Create Topic:

kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic your-topic-name-here
Enter fullscreen mode Exit fullscreen mode

Additional CLIs:
▪️ kafka-topics.sh --list --zookeeper zookeeper:2181: list all topics.
▪️ docker ps: to show running containers.

and that's all.

Next step: Set up two Node.js apps

1️⃣ create a package.json:

npm init

Enter fullscreen mode Exit fullscreen mode

2️⃣ install packages:

npm i express nodemon dotenv mongoose kafkajs
Enter fullscreen mode Exit fullscreen mode

3️⃣ Now we will configure our producer:

Image description
4️⃣ Add the following two lines: (to your index)

our producer

the result : 👇

producerresult

5️⃣ configure our consumer by adding this following in it:

our consumer

6️⃣ now we will start our scenario:

First of all, create your model, controller, and route. Inside the controller, you must add the following two lines:
import {run} from "../utils/producer.js";
run(response);

like this example below: 👇

User controller

7️⃣ update our producer:

producer

8️⃣ update our consumer:

Image description

THE RESULT 👏

from producer:

Image description

from consumer:

Image description


🪄Github-repo

In conclusion, Apache Kafka is a powerful distributed streaming platform that enables real-time data processing and stream processing applications. It can be integrated with Node.js, a popular JavaScript runtime, through the use of Kafka client libraries, allowing developers to build event-driven and data-driven applications using the two technologies together. Kafka is known for its scalability, high performance, and ability to handle high volumes of data, making it a valuable tool for building robust and scalable applications. However, it can also be complex to set up and maintain, and requires a good understanding of its architecture and configuration options in order to use it effectively. Overall, Kafka and Node.js can be a powerful combination for building efficient and scalable applications, and are worth considering for any project that requires real-time data processing or stream processing capabilities.

Top comments (8)

Collapse
 
mattbidewell profile image
Matthew Bidewell

great post!

curious, why have you added two brokers of the same address? (assuming theres a reason, if not just say :P)

Collapse
 
chafroudtarek profile image
Chafroud Tarek

The connection between producers and consumers in a Kafka cluster can be either to the same broker or to different brokers, depending on the configuration of the Kafka cluster and the requirements of the application. Factors that can influence this decision include the structure of the cluster, the goals of the application, and the desired level of scalability and performance.

Collapse
 
valdineisantos profile image
Valdinei dos Santos

Hi first congrats for your post.

I need get the last 10 messages ( desc ) for find a particular message. I tried with seek but the connection never ends.
Could you sent me a example to get "N" messages and disconnect if found or not this message

Collapse
 
jareechang profile image
Jerry

Good overview 👍

What use case are you using kafka for ?

Collapse
 
chafroudtarek profile image
Chafroud Tarek

Generally, I use it for communication between microservices, and thisi is a small example

Collapse
 
aminenafkha1 profile image
Nafkha-Med Amine

Good luck mate!🔥

Collapse
 
amirbenahmed profile image
Ben Amir

Hi Tarek thank you for sharing 🔥

Collapse
 
saifeddin1 profile image
Saifeddin Matoui

Thanks for sharing Tarek. That's so helpful