Message queues: SQS vs Kafka

Message queue battle royal SQS vs Kafka

Kafka and SQS are very popular message queues, for Kafka, it's a huge scale and operational pain, SQS with plug and play and far far less operational pain.

It becomes harder to choose which one to use as a developer, as a developer we will always choose the laziest approach which is SQS due to the soft process of just clicking here, and their boof you have a message queue.

When actually picking one on top of another I mean we all know that Kafka is a bit complex compared to SQS, in terms of having many brokers and partitions, replicas, and topics while in SQS you only need to specify your queue name and just publish and consume from it.

But there is more to it than the operational and development cost, what I noticed throughout my work with both is that Kafka is more oriented to handle the notion of many consumers & events is that the publisher needs to be aware of the consumer which starts having an unhealthy pattern of smart publisher that adjust the messages for each consumer, which is kinda bad as it makes the relation between publisher and consumer more coupled and strict.

To elaborate that more let's lead with a story,
So let's assume that we have a delivery system where we have many domains let's only focus on the order domain, at first we need to notify the user whenever the order is on its way, and we define that by a field status over the order's domain.

Now we can choose whether the order domain operation is a single queue/topic or each status is a different queue/topic.
Let's go with a single queue/topic for all order operations called orders

So far there is no difference between Kafka & SQS.

Time passes and we needed to start measuring how long it takes for the order to be delivered compared to our initial estimation which is again with the status thing but different consumer this time.

SQS vs Kafka

With Kafka, it's very easy to add a new consumer over our orders topic using the consumer group notion, but for SQS it would be a new queue.
This also makes the order service know who are the consumers and what they need so every time we need to do something different with the orders we will need to modify the orders service to publish to a different topic as well as integrate a new different queue config.

This is the reason why I don't like the usage of SQS with a domain that requires many events.

Let me know what you think of that in the comments below