Apache Kafka® vs. RabbitMQ™: Battle of the (Message) Brokers

Apache Kafka® and RabbitMQ™ are both popular messaging systems—each with its respective strengths, weaknesses, and ideal use cases. So, how should you decide what’s right for your applications and environment?

In this blog, we’ll break down the key differences between these two streaming platforms, describe the best use cases for each one, and give examples of some large enterprises who are leveraging one or the other (or both) for their applications.

Comparing architectures, protocols, and scalability

First off, it’s important to distinguish between these two platforms’ architecture. Kafka is a distributed broadcast-style message broker that does not rely on a message queue, but instead uses a write-only log. It saves messages to disk, appending the log for consumers to read, until it reaches capacity. Because the data persists, subscribers can go back in time to consume them (an example is a newsfeed, where some people post and others can read them as they are posted or scroll back in time to read them later). Kafka also lets users batch messages, which allows for higher throughput.

In contrast, RabbitMQ is a message broker designed to validate, route, and store communication between applications and services. Like a human translator, it lets these parties speak directly to each other, regardless of the languages they use or the platforms on which they are running. Unlike Kafka, RabbitMQ deletes messages after completing their delivery.

Kafka organizes messages in what it calls “topics.” A topic holds information that logically belongs together. An example might be “payment_processed.” Topics can be consumed by one or more consumers, which might live in different domains. This information may be consumed by “shipping_consumer,” for example, and later by “notification_consumer.” When a subscriber consumes a message, Kafka marks it as “read,” but, as mentioned earlier, does not delete the data.

RabbitMQ is designed to handle message-based communication between applications and is optimized for quick, reliable, message delivery to one consumer at a time, asynchronously. A great non-digital analog might be Starbucks. You place your order with the barista (analogous to a broker), who is processing multiple orders. You don’t receive your order immediately. Instead, you wait in a queue with other customers. When your order is complete, your cup is delivered with only one name on it—yours. In other words, this one barista serves multiple customers, delivering their individual orders, one at a time.

Both of these messaging platforms are scalable. Kafka’s distributed architecture gives it the advantage where it comes to throughput and processing speed. The difference isn’t trivial: Kafka can handle millions of messages per second. RabbitMQ—which allows you to create a structured architecture for publishers and consumers and can be configured to have nodes devoted to specific queues (each processed by only one consumer, as in the Starbucks example)—can handle hundreds of thousands of messages per second.

Which platform will meet your specific needs?

Rabbit MQ and Kafka shine in different areas, so your choice should depend on your organizations’ use cases.

Here are some of the best use cases for Kafka:

• Industries such as finance, healthcare, and e-commerce applications. All of these require high-volume, real-time data streams, which is where Kafka excels.
• Kafka is also perfect for fraud detection, user-behavior analytics, and predictive maintenance, since all three require real-time streaming analytics and need to process and store large volumes of data.
• Kafka can be used to build event-driven architectures, where events trigger actions in other systems or applications. This makes it a good fit for use cases such as microservices, IoT, and data pipelines.

Here are some of the best use cases for RabbitMQ:

• As explained earlier, RabbitMQ is a great choice for use cases that don’t require real-time processing. Asynchronous task queues—such as workflow automation, background job processing, and message-based task distribution—all fall into this category.
• RabbitMQ is optimized for reliable, asynchronous message delivery, making it a good fit for message-driven architectures where applications exchange messages with each other.
• RabbitMQ can be used to build distributed systems, where different components communicate with each other through message passing. This makes it a good fit for use cases such as chat applications, multiplayer games, and peer-to-peer networks.

The “big dogs:” who’s using what and why?

Some of the world’s largest enterprises are using Kafka for their event-driven, streaming, and messaging applications. The platform’s scalability, reliability, and real-time processing capabilities are critical to these organizations’ success.

Here’s who’s using Kafka and for what:

• LinkedIn: The company (which, by the way, originally created Kafka) uses Kafka extensively for data processing. This behemoth employs its platform to collect, process, and analyze real-time data from various sources across LinkedIn's infrastructure. One great example is its newsfeed, where millions of people post to the feed (the write-only/append functionality) and other people can read those posts either in real time or later, at their leisure.
• Netflix: Kafka underpins Netflix’s real-time streaming platform, which handles millions of events per second. The platform enables Netflix to process and analyze all this data in real time, providing insights into user behavior and enabling personalized recommendations.
• Uber: Among other real-time data-processing applications, Uber uses Kafka for its stream-processing framework, Flink. It enables the rideshare and delivery company to handle high volumes of real-time transactions. It also gives Uber insights into user behavior, traffic patterns, and pricing.
• Airbnb: Like Uber, Airbnb has its own stream-processing framework—StreamAlert. Kafka provides Airbnb with the real-time analytics it needs to understand its user behavior and provide personalized recommendations, among other things.
• Goldman Sachs: Kafka enables Goldman Sachs to process and analyze real-time market data. Its high-volume trading platform requires the scale that Kafka can provide to help its users make informed trading decisions.

Many large companies leverage RabbitMQ’s strengths for real-time communication, IoT platforms, microservices architecture, and mission-critical systems—in some cases, in addition to Kafka.

Here’s who’s using RabbitMQ and for what:

• Airbnb: Although it uses Kafka for StreamAlert, Airbnb uses RabbitMQ to power its messaging platform, which enables communication between hosts. It relies on RabbitMQ’s reliable message delivery and scalability, both of which are critical for the high volume of messages exchanged on Airbnb’s platform.
• Uber: Like Airbnb, Uber uses RabbitMQ, in addition to Kafka, to handle the real-time messaging exchanged between drivers and riders/customers. It also uses RabbitMQ for internal communication between different components of its systems.
• Siemens: Siemens uses RabbitMQ for MindSphere, a proprietary IoT platform for industrial applications. Mindsphere uses large volumes of messages from sensors and devices, which the company needs to process and analyze in real time.
• NASA: NASA’s telemetry system collects data from spacecraft and ground stations. RabbitMQ facilitates reliable and efficient message delivery, which are among its mission-critical applications.
• SoundCloud: RabbitMQ enables SoundCloud’s platform to scale its microservices architecture to handle high volumes of requests. It relies on RabbitMQ for reliable message delivery and enables communication between different components of the system.

It’s a multi-platform world

In sum, there are definite differences between Kafka and RabbitMQ that make them useful for specific applications. And as you can see, they’re not mutually exclusive, either.

What should you choose?

• Do you have a need for a messaging system that involves one message per person, like Starbucks? You should choose RabbitMQ.
• Do you stream large volumes of data, where the data is the same but there are multiple receivers, like the Netflix example? Kafka is your best option.
• Do you need to persist your data? Go Kafka.
• Do you have peer-to-peer networks? RabbitMQ’s for you.

Does your company use Kafka, RabbitMQ, or both? Let me know how and why in the comments. And don't forget to connect with Outshift on Slack!