How it all started

As a Kafka guy, when I'm helping teams learning about how to setup Kafka, there's always one recurring element: how do you monitor Kafka ?

Kafka is a wonderful piece of machinery, but it's complex to manage. You have to monitor (at minimum) 2 products, Zookeeper (3 or 5 instances, in typical installation) and the Kafka brokers (3 or 4 instances, usually, but as an architect would say: "it depends of the context").

For Zookeeper, to start you can just say: is it alive ? Does it respond to requests ? And It's usually fine. But for the brokers, knowing they are running doesn't tell you the whole story. You have to understand how they are running: is your cluster healthy ? Is it serving requests ? How full is your request queue ? etc.

Once you monitor the brokers, you should also monitor the client application. How are they performing in their interaction with Kafka ?

And once you have done that, you have to consider monitoring all other parts of the Kafka ecosystem : Kafka connect, Kafka streams ecosystem, Schema registry, etc.

The amount of things you have to understand about Kafka to monitor it can be overwhelming. For us, at Zenika, the best thing is to start with some dashboards that can tell you a story.

What we want as Zenika's Kafka guys is to provide a way for teams to easily bootstrap a generic monitoring setup, learn from it, and make their own dashboard and alerting once they understand how all of this works.

Our open source project

With a colleague, we decided it was time to make an open source project that would show default dashboards for Kafka. We started the project kafka-monitoring as a draft of what it would look like.

We worked together on the goal: providing Grafana dashboards that teams could easily copy and setup in their context. We wanted to address all Kafka ecosystem products: producer, consumers, connectors, stream apps (and if we had the time, KSQL, Schema Registry, Rest Proxy ?).

A docker-compose file was the way to go, which would explain some of the best practices in Kafka configuration (like a default replication.factor and min.insync.replicas).

As big fan of agile, we started with the most valuable thing: the metrics of the Kafka brokers. We worked on a first Grafana dashboard.

StreamThoughts already thought of this

A huge ecosystem like Kafka has dozen of open source project. Surely some Kakfa dashboards already existed! That's true, but how do you navigate this sea of projects ? How do you know which dashboards are reliable, and which dashboards are up to date ?

It's when we encountered StreamThoughts' project, that we knew that instead of doing our own work, we would contribute to existing work. We knew of StreamThoughts because it's co-founder, Florian Hussonnois, is an ex-colleague of ours.

That's the magic of open source: finding people that want to solve the same problem, and making them work together!

An exciting opportunity

We forked StreamThoughts' project and are starting to work on some contribution! I'm excited to see where this will bring us.