DEV Community

Cover image for Apache Kafka - Fundamentals, Use cases and Trade-Offs
Dhruvesh Patel
Dhruvesh Patel

Posted on


Apache Kafka - Fundamentals, Use cases and Trade-Offs

This post will focus on open source and popular Apache kafka platform related brief fundamentals, Use Cases and Trade-offs with other popular messaging technologies.

Kafka Fundamentals - Brief

Apache kafka is an event streaming platform. It provides distributed, high-throughput, low-latency, fault-tolerant platform for handling real-time data feeds - known as events.

It can be used in different ways due to various capabilities combined in Kafka.

  • Message System - to Publish and Subscribe events
  • Stream Processing System - to Process steams of events
  • Storage System - to Store log of events reliably (forever, if needed)

Kafka has topics, partitions, producers and consumers, which are basics for any messaging system. For more details on Kafka 101, please refer this link, where it is very nicely explained.

Kafka has following message / event delivery guarantees between producer and consumer.

  • At most once โ€” Messages may be lost but are never redelivered.
  • At least once โ€” Messages are never lost but may be redelivered.
  • Exactly once โ€” Each message is delivered once and only once.

Kafka Message Delivery Guarantees

Kafka Use Cases

For anything that deals with events and real-time data, Kafka is the way to go to process events and get insights - all in real-time. Kafka supports wide variety of use cases due to large ecosystem. Some of the example use cases are as below.

  • Financial Services - Real-time fraud detection and faster payments
  • Automobile - Real-time data cluster for Anomaly detection, Alerts and Personalization for better driving experience
  • Healthcare - Real-time health sensors data to get better insights to serve patients and save lives

Kakfa can be used as backbone to integrate various systems - be it data analytics use case or microservices communications in an event-driven architecture. Following diagram shows kafka as backbone for various systems integration.

Kakfa - Use Cases and Systems

Kafka Trade-offs with other messaging systems

For basic, single machine, message oriented, point-to-point systems communication, classical messaging suits with RabbitMQ or ActiveMQ.

But for any event-streaming, where there is event-based communication with multiple consumers and need ordered events (either same events to multiple consumers or specific events to specific consumer group), then Kafka is best suited.

Kafka vs Legacy messaging systems

Apart from being messaging system, Kafka can be leveraged as stream Processing System (to Process steams of events) and Storage System (to Store log of events reliably - forever, if needed). It also has lots of other features in Kafka ecosystem, such as kafka connectors and kafka streams (please refer kafka documentation for more details). So for event-driven, real-time systems - Kafka is most matured and complete ecosystem.

Enjoy building event-driven systems with Kafka !

Please like, share and comment your views/ opinion.

Disclaimer - This post is my personal opinion and does not reflect those of any of my past, present or future employers or affiliations.

Top comments (18)

ziggo profile image

Please, can you explain better when to use Kafka instead of Rabbit MQ? I haven't really understand, you said that Kafka is used for log storage, but in the official tutorial of RabbitMQ there is also an example about log storage. What are the differences?

dhruvesh_patel profile image
Dhruvesh Patel

Kafka uses a distributed commit log as its storage layer. Writes are appended to the end of the log and Reads are sequential starting from an offset. This suites best for event streaming use cases.

Conversely, RabbitMQ uses index-based storage systems. These keep data in a tree structure to provide the fast access necessary for acknowledging individual messages, but fast individual reads comes at the cost of write overhead, which leads to either decreased write throughput and increased write latency compared to a log.

RabbitMQ is designed to store messages for a short period of time only. Kafka can retain messages indefinitely.

Hope this helps.

shreya1310 profile image

Very helpful article ๐Ÿ‘

dhruvesh_patel profile image
Dhruvesh Patel

Thanks for kind words.

levirs565 profile image
Levi Rizki Saputra

Good explanation.๐Ÿ‘

dhruvesh_patel profile image
Dhruvesh Patel

Thanks for the feedback.

prabhat36539454 profile image
Prabhat Kumar

Always a great learning to go through your article.

dhruvesh_patel profile image
Dhruvesh Patel

Glad to help.

chiragks profile image
Chirag Shah

Nice article . Always enjoyed your post. Keep share like this.

dhruvesh_patel profile image
Dhruvesh Patel

Thanks for kind words.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.

Timeless DEV post...

Git Concepts I Wish I Knew Years Ago

The most used technology by developers is not Javascript.

It's not Python or HTML.

It hardly even gets mentioned in interviews or listed as a pre-requisite for jobs.

I'm talking about Git and version control of course.

One does not simply learn git