OpenTelemetry KafkaJS Instrumentation for Node.js

TL;DR — Our JS OpenTelemetry plugin for kafkajs, is available here.

This article describes this plugin for the kafkajs package and our thought process behind it.

Some Background

OpenTelemetry is a CNCF project, which, among other things, enables the collection of distributed traces.

At Aspecto, we use OpenTelemetry at the core of our product.

While implementing it in our backend, we found a few plugins that were missing, especially when dealing with asynchronous communication.

One of them was kafkaJS.

We took this opportunity to give back to the community and developed it ourselves.

The Plugin

This plugin allows you to track all Kafka interactions in your collected traces, which means, you get a more comprehensive view of your application behavior when using Kafka as a message broker.

The kafkajs plugin captures producer and consumer operations and creates spans according to the semantic conventions for Messaging Systems.

• Each message being produced and consumed is represented by a span with attributes such as messaging.destination(topic name).

• Context is propagated from producers to consumers. When a message is sent to Kafka, the trace will reveal which services consume it and what other cascading operations happen down the pipe.

• Batch operations can aggregate multiple messages into a single batch and receive\process them together. This is handled in the plugin according to the specification.

• The plugin can be extended with hooks, which enable users to run custom logic for adding span attributes depending on the Kafka message.

The above screenshot shows an example of a producer application named kafka-producer, that exposes an HTTP endpoint (first line), route it in express (second line), and produce two messages to a Kafka topic named test, which are then consumed by another application called kafka-consumer .

As mentioned above, kafkajs was one of the missing plugins we found and as you are reading this we are working to add more plugins.

Feel free to reach out to us with any questions as we are very much invested in OpenTelemetry and the OpenTelementry community.