This post will focus on open source and popular Apache kafka platform related brief fundamentals, Use Cases and Trade-offs with other popular messaging technologies.
Kafka Fundamentals - Brief
Apache kafka is an event streaming platform. It provides distributed, high-throughput, low-latency, fault-tolerant platform for handling real-time data feeds - known as events.
It can be used in different ways due to various capabilities combined in Kafka.
- Message System - to Publish and Subscribe events
- Stream Processing System - to Process steams of events
- Storage System - to Store log of events reliably (forever, if needed)
Kafka has topics, partitions, producers and consumers, which are basics for any messaging system. For more details on Kafka 101, please refer this link, where it is very nicely explained.
Kafka has following message / event delivery guarantees between producer and consumer.
- At most once — Messages may be lost but are never redelivered.
- At least once — Messages are never lost but may be redelivered.
- Exactly once — Each message is delivered once and only once.
Kafka Use Cases
For anything that deals with events and real-time data, Kafka is the way to go to process events and get insights - all in real-time. Kafka supports wide variety of use cases due to large ecosystem. Some of the example use cases are as below.
- Financial Services - Real-time fraud detection and faster payments
- Automobile - Real-time data cluster for Anomaly detection, Alerts and Personalization for better driving experience
- Healthcare - Real-time health sensors data to get better insights to serve patients and save lives
Kakfa can be used as backbone to integrate various systems - be it data analytics use case or microservices communications in an event-driven architecture. Following diagram shows kafka as backbone for various systems integration.
Kafka Trade-offs with other messaging systems
For basic, single machine, message oriented, point-to-point systems communication, classical messaging suits with RabbitMQ or ActiveMQ.
But for any event-streaming, where there is event-based communication with multiple consumers and need ordered events (either same events to multiple consumers or specific events to specific consumer group), then Kafka is best suited.
Apart from being messaging system, Kafka can be leveraged as stream Processing System (to Process steams of events) and Storage System (to Store log of events reliably - forever, if needed). It also has lots of other features in Kafka ecosystem, such as kafka connectors and kafka streams (please refer kafka documentation for more details). So for event-driven, real-time systems - Kafka is most matured and complete ecosystem.
Enjoy building event-driven systems with Kafka !
Please like, share and comment your views/ opinion.
Disclaimer - This post is my personal opinion and does not reflect those of any of my past, present or future employers or affiliations.