A Gentle Introduction to Kafka Consumer Group

You may have heard about Kafka if you are studying for a system design interview or dealing with data-intensive applications. Even if you don't work with distributed systems before, you probably have heard of "Kafka" during a water cooler chat with your colleagues.

In the system design interview, Kafka was a magic box that solved all reliability and scalability issues. In the industry, Kafka was trusted by Fortune 100 companies to be the backbone of their scalable infrastructure.

I got to work on Kafka for the past six months, trying to scale our event-driven notification pipeline. I realized many terminologies differ from AWS Kinesis, GCP Pubsub, and Kafka. However, Kafka is a versatile tool that all software engineers should consider when scaling their systems to millions of users.

In this article, I am giving you a quick introduction to Kafka, specifically on what is a Consumer group, because it is a new and fresh perspective coming from someone who knows AWS Kinesis. This article can be a very brief introduction to the Kafka component. Still, I will dive deep into the Consumer group and give you some examples of how the Consumer group fits into a system case study.

Now without further ado, let's dive in.

What are the High-Level components of Kafka?

Like any other message queue system, at a high level, Kafka has a producer, broker, and consumer.

Broker

When you deploy a Kafka Broker, you don't just deploy a single machine. You deployed multiple machines, which consist of clusters. With all those machines, Kafka uses a distributed config system, Zookeeper, to manage and coordinate its node and cluster. ZooKeeper will notify the leader in the cluster to ensure that it is not dead. If the leader dies, ZooKeeper will help elect a new leader to ensure the broker is highly available.

I often got a question from a system design interview when designing applications with Kafka, "If everything needs to go through Kafka, isn't Kafka a single point of failure?" The answer is No, because, in practice, you will need to spin up multiple replicas on a broker.

Consumer

Consumers in a Kafka read data and consumer messages from Topics they subscribed to. The topic is a way to separate each record into categories. Multiple consumers can consume from that category. Each topic will have its partition, which helps distribute the load and increase availability and throughput.

Consumers keep track of their offsets in a special topic called __consumer_offsets. Thus, record offset helps the consumer to pick up where they left off if they die. Thus, it doesn't need to consume the record from the beginning of the offset. Kafka employs an approach of a dumb pipeline, smart clients meaning that Kafka brokers don't know anything about consumer offsets.

Producer

Kafka producers message directly to the corresponding topic partitions. It also serializes, compresses, and loads balance data among brokers through partitioning.

What is the Consumer Group in Kafka?

When you have a system that consists of many producers to many consumers, you will require a way to have a topic consumed by multiple consumers without stepping on each other's toes. Kafka achieved that by separating the consumers into its group. Kafka uses group id to identify all the consumers in the same group. When a consumer is in the same group, every record from the topic will only be delivered to that consumer.

Consumer Group Gotchas

Remember that topic consists of one or more partitions? Aside from the purpose of increasing throughput and availability, Kafka also ensures that only one consumer consumes each partition in that group.

When a consumer is started, it will join a consumer group (this happens under the hood).

Now, what happens if you increase the consumer to two?

Each consumer will be consuming one partition.

Again, what happens if you have more consumers in the group than the partition provided by the broker?

That's right! You guessed it. That one consumer will be sitting there idle.

If you want to increase the consumer count, you will need to plan accordingly by also increasing the number of partitions. Thus, the partition is a unit of parallelism from the consumers' perspective. Why?

You know the reason now. 😏

Now that you know briefly what a consumer group is, let's go into some real-life examples of using a consumer group to achieve some non-functional requirements from business ask.

Example and Use Cases of Changing Consumer Group

"My message needs to be Processed in Order."

We have order events, ORDER_RECEIVED, ORDER_CREATED, ORDER _CANCELED. Kafka Producer will publish the event into a single Order Topic distributed across the partitions. On the other end, multiple consumers will poll messages from the topic in parallel and process each one of the events. We want the message to be processed in order with such a system. An ORDER_CANCELED event on an ordered 123 can only be processed after ORDER_CREATED since it makes no sense.

How can we guarantee message ordering in Kafka?

Remember that each partition can only be consumed by one consumer? That is key! Because partition and consumer are 1:1 relationships, putting the same message into the same partition will consume it by one consumer.

The message partition key determines a message partition. In the above scenario, to ensure the message is processed in order, we can create a hash as the partition key of the producer so that each time the producer sends a record, it will go to the same partition. The partition key can be based on the order since the producer will produce different event types based on a single order Id. Therefore, we ensure that the same order will always be at the same partition to preserve message ordering.

"I want also to Process multiple things in this Event, not just XYZ."

The business use case above is that we want to broadcast the event to multiple consumers. One example is that we want to notify the user that the payment has been authorized and update the order table to mention that the order has been paid. We will require multiple consumers to receive the same event topic. We can do this by putting order and notification consumers in 2 different groups. This way, the order and notification consumers will receive the same amount of message records and won't cause any skipped messages.

"My App often received multiple duplicate notifications."

The problem is the side effect of trying to have multiple consumer instances consume multiple records and process duplicate events.

For example, we don't want multiple instances of notification service to process the payment record and send multiple "Your payment has been received" emails to the customer.

We want exactly one of the instances to process each record. We would put all the notification service instances in one consumer group to do this. This way, we can ensure that one record will always go to only one instance within that group.

Another duplication problem occurs when the consumer re-processed the event again because of client failure while committing its record offset. Kafka lets you configure how consumers want to commit their records. You can set auto-commit to true, and your consumer will automatically commit the message once received. In this case, it is At-most-once. One of the disadvantages of at-most-once is that if the consumer dies before processing the data, that data can be gone forever.

To fence off duplicates, the producer includes a Kafka-assigned ID and a monotonically increasing sequence number when sending messages. Kafka rejects the message if there is already a committed message from the same producer(identified by Kafka-assigned ID) with an equal to the higher sequence number. To ensure exactly-once delivery semantics, we must guard the message processing and offset storage to be at the same transaction. That transaction can be a traditional database transaction that stores both the output of the message and the updated offsetting the same commit. Kafka has transaction semantics in publishing multiple topics, which allows consumers to store the output and offset atomically in two recipient Kafka topics.

Recap

Kafka is a very powerful tool that is scalable and reliable. One of its interesting features is the consumer group, which groups your consumer into the same group id to effectively load-balanced load across consumer instances.

If you cannot remember everything, here are the key takeaway of the consumer group:

1. Partition and consumer inside the consumer group is a 1:1 relationship. Thus, if you want to scale your consumer, you must also scale your partitions.

2. A partition is a unit of parallelism in Kafka. Each consumer can process inside the consumer group in Parallel.

3. You can create a hash during publishing your message to ensure certain message goes to a specific partition.

I hope you learn something about the Kafka consumer group in this post. As always, feel free to post your questions below if you have any. Have a great week ahead.

Thanks for reading Path To Senior! Subscribe for free to receive new posts and support my work.

💡 Want more actionable advice about Software engineering?

I’m Edward. I started writing as a Software Engineer at Disney Streaming Service, trying to document my learnings as I step into a Senior role. I write about functional programming, Scala, distributed systems, and careers-development.

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/