loading...
Cover image for Kafka Microservice using Nest Js

Kafka Microservice using Nest Js

kannndev profile image Kannan ・3 min read

Hello All,

Hope you are doing well.
In this article, we are going to learn about how to set up a microservice using NestJs and Kafka.

Why did I start writing this?
I recently wanted to set up one and I was struggling to find a good example. After hours of googling, I was able to. So to make life easier for my fellow developers here I am.

Note: In this, I will not be explaining about Kafka or NestJs. So some basic knowledge on those would be helpful while implementing it.

Let's Get Started!!!

To install on windows/ubuntu please refer:
windows
ubuntu

To install Kafka on Mac using Homebrew:

$ brew cask install java
$ brew install Kafka

To start Zookeeper:

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

Note: Zookeeper should be always started before starting the Kafka server.

To start Kafka:

kafka-server-start /usr/local/etc/kafka/server.properties

Lets check whether Kafka is working properly

To create a topic:

kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic my-first-topic

To create a Producer console:

kafka-console-producer --broker-list localhost:9092 --topic my-first-topic

To create a Consumer console:

kafka-console-consumer --bootstrap-server localhost:9092 --topic my-first-topic --from-beginning

Now Producer console will be able to accept user inputs. Whatever we type on the Producer console should be visible in the Consumer console.

Let's write some code now

Let's create a NestJs application

nest new kafka-consumer

Once the application is created we need the microservices module and Kafka client library.

npm i --save @nestjs/microservices
npm i --save kafkajs

Now we need to update our main.ts to accept Kafka Transport and provide Kafka configurations

// main.ts

import { NestFactory } from '@nestjs/core';
import { Transport, MicroserviceOptions } from '@nestjs/microservices';

import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(AppModule, {
    transport: Transport.KAFKA,
    options: {
      client: {
        brokers: ['localhost:9092'],
      },
      consumer: {
          groupId: 'my-kafka-consumer',
      }
    }
  });

  app.listen(() => console.log('Kafka consumer service is listening!'))
}
bootstrap();

In app.controller.ts we are listening to our topic.

import { Controller } from '@nestjs/common';
import { MessagePattern, Payload } from "@nestjs/microservices";
import { AppService } from './app.service';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) { }

  @MessagePattern('my-first-topic') // Our topic name
  getHello(@Payload() message) {
    console.log(message.value);
    return 'Hello World';
  }
}

Start our server

npm start

Now let's go back to the producer console and send a message now it should be logged in our application console.

Voila!. Now we have set up our Kafka microservice successfully.

In case you want to set up producer inside a nest js application Please follow along.

Create an application and install the necessary dependencies

nest new kafka-producer
npm i --save @nestjs/microservices
npm i --save kafkajs

In app.controller.ts will set up our producer:

import { Controller, Get } from '@nestjs/common';
import { Client, ClientKafka, Transport } from "@nestjs/microservices";
import { AppService } from './app.service';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) { }

  @Client({
    transport: Transport.KAFKA,
    options: {
      client: {
        clientId: 'kafkaSample',
        brokers: ['localhost:9092'],
      },
      consumer: {
        groupId: 'my-kafka-consumer' // Should be the same thing we give in consumer
      }
    }
  })
  client: ClientKafka;

  async onModuleInit() {
    // Need to subscribe to topic 
    // so that we can get the response from kafka microservice
    this.client.subscribeToResponseOf('my-first-topic');
    await this.client.connect();
  }

  @Get()
  getHello() {
    return this.client.send('my-first-topic', 'Hello Kafka'); // args - topic, message
  }
}

Quit the producer console and consumer console that we started earlier if not already.

Start our server

npm start

Now open any Rest API client and hit
GET: http://localhost:3000/

Now we can see the Hello World as our response sent from Kafka microservice.

You can find the source code here:
Consumer Sample
Producer Sample

P.S This is my first blog post. Please feel free to give feedbacks.

Posted on by:

kannndev profile

Kannan

@kannndev

Full Stack Javascript Developer

Discussion

pic
Editor guide