Using Apache Kafka with Node.js: A Tutorial on Building Event-Driven Applications

Apache Kafka is a powerful, open-source event streaming platform used to build distributed, real-time applications. When combined with Node.js, a lightweight JavaScript runtime, developers can build event-driven applications in a fraction of the time. In this tutorial, we'll explore how to use Apache Kafka and Node.js together, and how to create a simple event-driven application.

Prerequisites

Before we dive into using Apache Kafka and Node.js, you’ll need to make sure you have the following prerequisites installed on your machine:

• Node.js version 12+
• Node Package Manager version 6+
• Apache Kafka 2.4+

You will also need a basic understanding of Node.js, JavaScript, and Apache Kafka. You can find more information about these topics in the following blog posts:

• Building a Microservice Architecture with Node.js, TypeScript, and gRPC
• Getting Started with Golang: A Beginner's Guide to Writing Go Code
• Creating a RESTful API with Node.js and MongoDB
• Building a Real-Time Chat Application with Node.js, Socket.IO, and Redis
• Introduction to TypeScript: Adding Types to JavaScript
• Using PostgreSQL with Node.js: A Step-by-Step Guide
• Building a Full-Stack Web Application with Node.js, Express, React, and MySQL
• Getting Started with MongoDB: An Overview of NoSQL Databases
• Creating a GraphQL API with Node.js and PostgreSQL
• Using gRPC with Node.js: A Tutorial on Building High-Performance APIs
• Building a Serverless Application with Node.js and AWS Lambda
• An Introduction to Docker for Node.js Developers
• Building a RESTful API with TypeScript and Express
• Migrating from MySQL to PostgreSQL: A Step-by-Step Guide

Setting up the Environment

Before we dive into the tutorial, you'll need to set up your environment. To begin, install Apache Kafka with the command:

$ sudo apt-get install kafka

Next, start up the Kafka service with the commands:

$ sudo service kafka start

Now that you have Kafka installed and running, let’s move on to installing the Node modules we’ll need for our application. You’ll need kafka-node and its dependencies. To install these, simply run the following command:

$ npm install kafka-node

Writing the Code

Now that we have all of the prerequisites installed and our environment set up, it’s time to start writing our code. To begin, create a new file called producer.js, and add the following code:

const kafka = require('kafka-node');

// Create a new producer
const producer = new kafka.Producer();

// Connect to Kafka broker
producer.connect();

// Send a message to Kafka
producer.send(['my-topic'], [{message: 'Hello, Kafka!'}], (err, data) => {
  if (err) console.error(err); 
  else console.log(data);
});

This code creates a new producer, connects to the Kafka broker, and sends a message to the my-topic topic.

Next, create a new file called consumer.js, and add the following code:

const kafka = require('kafka-node');

// Create a new consumer
const consumer = new kafka.Consumer();

// Connect to Kafka broker
consumer.connect();

// Listen to the 'my-topic' topic
consumer.on('message', (message) => {
  console.log(message);
});

This code creates a new consumer, connects to the Kafka broker, and listens for messages on the my-topic topic.

Now that we’ve written the code for both the producer and the consumer, let’s move on to setting up an event-driven application.

Building the Event-Driven Application

To create an event-driven application, we will use Node.js to listen for events, and Apache Kafka to send and receive the events. We will combine the producer and consumer from the previous section to create a simple event-driven application.

First, create a new file called app.js and add the following code:

const kafka = require('kafka-node');

// Create producers
const producerA = new kafka.Producer();
const producerB = new kafka.Producer();

// Connect to Kafka broker
producerA.connect();
producerB.connect();

// Set up event listener
const messageListener = (message) => {
  if (message === 'A') producerA.send(['my-topic'], [{message: 'Hello, Kafka!'}], (err, data) => {
    if (err) console.error(err); 
    else console.log(data);
  });
  else if (message === 'B') producerB.send(['my-topic'], [{message: 'Hello, Kafka!'}], (err, data) => {
    if (err) console.error(err); 
    else console.log(data);
  });
};

// Create consumer
const consumer = new kafka.Consumer();

// Connect to Kafka broker
consumer.connect();

// Listen to the 'my-topic' topic
consumer.on('message', messageListener);

This code creates two producers, connects them to the Kafka broker, and sets up an event listener. The event listener listens for events on the my-topic topic, and sends a message if one of the events is received.

Now that we’ve written the code for the application, let’s move on to running the application.

Running the Application

To run the application, you’ll need to start the Apache Kafka server. To do this, run the command:

$ sudo service kafka start

Once the Apache Kafka server is running, you can start the application with the command:

$ node app.js

Now that the application is running, we can send messages to the my-topic topic and see the results in the console. To do this, run the command:

$ kafka-console-producer --broker-list localhost:9092 --topic my-topic

This command allows us to send messages directly to the my-topic topic. When you enter a message, you should see the results in the console.

Conclusion

In this tutorial, we’ve explored how to use Apache Kafka and Node.js together to create an event-driven application. We’ve seen how to set up the environment, write the code, and run the application. With Apache Kafka and Node.js, developers can quickly and easily create powerful distributed applications.