In Grafana Dashboard, when we need to create a new Dashboard to show how many messages has produced by Kafka topics (with Kafka exporter + Prometheus) in a period, we can specify every topic as single metric like this below image

It works fine for sure! But what if you have more than 2 topics and your topics actually have the same pattern which could be matching by regex. The next time, if any new topic gets created, the dashboard will automatically reflect and show on the Grafana dashboard, then you don't have to do it as the “old” way.

Here is the query from image above

sum(delta(kafka_topic_partition_current_offset{job="kubernetes-service-endpoints",kubernetes_name="prod-global-kafka-exporter", topic=~"{{regex1}}", topic!~"{{regex2}}"}[1m])) by (topic)

• {{regex1}}: Replace with a regular expression to match any topic we want to show. Example: prod-myservice.*

• {{regex2}}: Replace with a regular expression to match any topic we want to avoid showing. Example: prod-myservice.+deprecated.*

A useful web tool to check if the regex could work is https://regex101.com/.

Referenced sources

• Grafana: https://grafana.com/
• Prometheus: https://prometheus.io/
• Kafka exporter for Prometheus: https://github.com/danielqsj/kafka_exporter

Happy monitoring!