AWS Kafka with Node.js

To use AWS Kafka with Node.js, use the kafkajs library which provides a client for Kafka in Node.js. Here how to use kafkajs with AWS Kafka:

Install the kafkajs library using npm:

npm install kafkajs

Create a new instance of the Kafka client with the AWS configuration:

const { Kafka } = require('kafkajs')
const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka-1.amazonaws.com:9092', 'kafka-2.amazonaws.com:9092'],
  ssl: true,
  sasl: {
    mechanism: 'scram-sha-256',
    username: 'my-username',
    password: 'my-password',
  },
})

Create a producer to send messages to a topic:
javascript

const producer = kafka.producer()
await producer.connect()
await producer.send({
  topic: 'my-topic',
  messages: [
    { value: 'Hello Kafka!' },
  ],
})
await producer.disconnect()

Create a consumer to receive messages from a topic:

const consumer = kafka.consumer({ groupId: 'my-group' })
await consumer.connect()
await consumer.subscribe({ topic: 'my-topic', fromBeginning: true })
await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    console.log({
      topic,
      partition,
      offset: message.offset,
      value: message.value.toString(),
    })
  },
})
await consumer.disconnect()

That's it! You should now be able to use AWS Kafka with Node.js using the kafkajs library.