Users of Apache Kafka® have varying opinions on so many things—it’s what makes the community diverse and interesting. But regardless of what your policy on schemas is (they’re obviously good) or whether you believe that adding partitions to a Kafka Topic is okay (it’s definitely not), I’ll bet you can still find some common ground with your fellow users.
Case in point: pretty much everyone that works with Kafka can agree that auto.offset.reset is quite possibly the worst named Kafka configuration out there.
Driven by my frustration as a user and my own curiosity as a developer, I decided to dive into this configuration and try to understand it and its awful name, once and for all.
But first…
Some background on auto.offset.reset
I strongly believe that this particular origin story will be helpful for anyone, regardless of whether they’ve been using Kafka for years or days. So let’s start by getting everyone on the same page.
Kafka Consumers. You know them, right? They’re pretty useful in helping you to consume messages from Kafka Topics. As you read messages from a Topic, every so often, the Consumer will record the last message that it saw and processed using the offset of that message. This is really helpful so that, if the Consumer goes down for any reason, you can rest assured that, when it comes back online, the Consumer can use that stored offset to pick back up from its last committed position.
Once our Consumers are up and running, offsets and Consumers go hand-in-hand. But what about before the Consumer has started running? What about brand new Consumers that have never seen a single message from the Kafka Topic? That’s where auto.offset.reset comes in.
Consumers that don’t have an offset available to use still need to know where to start reading from in the Topic. To put it simply, auto.offset.reset is a Kafka Consumer configuration, which can be set to either earliest or latest (or None), that defines where a Consumer should begin reading from in the Kafka Topic when it doesn’t have any other valid offsets to start from.
That last bit is really important, and it’s what trips folks up when they first try to use and understand this configuration parameter.
So, what is in a name?
The biggest issue that folks appear to have with auto.offset.reset is its name, which is a little disappointing because one would think that a three-word configuration would have the potential to be short, sweet, and to-the-point. Alas, sasl.oauthbearer.jwks.endpoint.retry.backoff.ms appears to both be more descriptive and far less problematic than auto.offset.reset.
So what does auto.offset.reset imply? What does someone using this configuration for the first time assume, erroneously, that it does? I’ve heard a number of variations, but generally folks assume that it automatically resets the stored Consumer offset to earliest or latest.
That… would make sense, right? Except that we all know the frustration, the pain, the anguish, when we set auto.offset.reset=earliest and realize that our Kafka Consumer is not, in fact, automatically reading from the beginning of the Topic, nor did it reset our offsets. The reality is that, if there's a valid offset, auto.offset.reset doesn't do a single thing.
So what gives? Where is this disconnect with auto and reset coming from?
Auto vs. Manual
Let’s start with auto. Take a look at the rest of the Kafka Consumer configurations that have auto in them; they all, truly, have to do with handling something automatically for the Consumer.
What does auto.offset.reset do for us automatically? It certainly doesn't appear to be to overriding Consumer offsets automatically. But to really make sense of what auto.offset.reset does for us behind the scenes, we have to think about the manual version of this offset-setting process.
A Consumer without an offset
Consider what would happen if you didn’t want to set auto.offset.reset, but you had a brand new Consumer (or a Consumer whose offsets you've deleted).
If you read the auto.offset.reset documentation closely, you’ll find that you don’t actually have to set it. But, when auto.offset.reset=None and you have no stored offsets, when your Kafka Consumer tries to poll the Topic, it will throw an exception. This implies that, oops, our Kafka Consumers need initial offsets to actually read from the Topic.
Setting offsets manually
A brand new Consumer or a Consumer without offsets doesn’t have to rely on auto.offset.reset, though. You have the option to manually set your offsets using Consumer.seek()—as well as the related methods Consumer.seekToBeginning() and Consumer.seekToEnd().
seek() is an in-memory operation that takes in a TopicPartition and an offset and manually overrides the offset that the Consumer has for that partition—even if it currently has no offset for that partition.
Never heard of Consumer.seek()? That’s probably because it’s mostly used in manual one-off scripts—at least, that’s been my experience with it.
Now that you know about the existence of a way to manually set Consumer offsets, doesn’t the auto in auto.offset.reset make just a little bit more sense? That is, until we think a bit more about the fact that it doesn’t reset anything…
Reset, but only just sometimes
Okay, bear with me, here.
Let’s go back to the official Apache Kafka auto.offset.reset documentation and focus on a bit that I’ll bet you missed the first time around:
What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted)
Bingo.
Unless you have infinite retention set up, data doesn’t live in a Kafka Topic forever. Eventually, it’ll be deleted or compacted. If, for some reason, our Consumer is inactive and the message with the last offset that that Consumer has seen is deleted, then suddenly we have an invalid offset for that Topic—which, when you think about it, is just as bad as having no offset at all because having no offset is just a special case of an invalid offset...
Gee, it sure would be nice if our Consumer knew where to start from when it comes back online. If we have auto.offset.reset set up, automatically the Consumer would reset its invalid offset to one of earliest or latest, meaning that we don’t have to worry about the Consumer throwing an exception.
Automatic and resetting
So where are we with auto.offset.reset now?
In the event that there aren’t any valid offsets available for a Kafka Consumer—whether that means no offsets at all or a stale offset from a deleted message—auto.offset.reset defines how to automatically reset the invalid offsets and allow the Consumer to continue operating.
This probably isn’t what you wanted to hear, but, after all of that, the name auto.offset.reset does a pretty decent job describing exactly what it does. Kafka users everywhere just needed a little bit more context to understand why this frustrating configuration is called what it’s called.
So what do you think? Is this explanation enough to make you just a little bit less frustrated at auto.offset.reset? If not, what would you call it?
Top comments (0)