When using asynchronous communication for Microservices, it is common to use a message broker. A broker ensures communication between different microservices is reliable and stable, that the messages are managed and monitored within the system and that messages don’t get lost. There are a few message brokers you can choose from, varying in scale and data capabilities. This blog post will compare the three most popular brokers: RabbitMQ, Kafka and Redis.
Microservices Communication: Synchronous and Asynchronous
There are two common ways Microservices communicate with each other: Synchronous and Asynchronous. In a Synchronous communication, the caller waits for a response before sending the next message, and it operates as a REST protocol on top of HTTP. On the contrary, in an Asynchronous communication the messages are sent without waiting for a response. This is suited for distributed systems, and usually requires a message broker to manage the messages.
The type of communication you choose should consider different parameters, such as how you structure your Microservices, what infrastructure you have in place, latency, scale, dependencies and the purpose of the communication. Asynchronous communication may be more complicated to establish and requires adding more components to stack, but the advantages of using Asynchronous communication for Microservices outweigh the cons.
Asynchronous Communication Advantages
First and foremost, asynchronous communication is non-blocking by definition. It also supports better scaling than Synchronous operations. Third, in the event Microservice crashes, Asynchronous communication mechanisms provide various recovery techniques and is generally better at handling errors pertaining to the crash. In addition, when using brokers instead of a REST protocol, the services receiving communication don’t really need to know each other. A new service can even be introduced after an old one has been running for a long time, i.e better decoupling services.
Finally, when choosing Asynchronous operations, you increase your capability of creating a central discovery, monitoring, load balancing, or even policy enforcer in the future. This will provide you with abilities for flexibility, scalability and more capabilities in your code and system building.
Choosing the Right Message Broker
Asynchronous communication is usually manages through a message broker. There are other ways as well, such as aysncio, but they’re more scarce and limited.
When choosing a broker for executing your asynchronous operations, you should consider a few things:
- Broker Scale – The number of messages sent per second in the system.
- Data Persistency – The ability to recover messages.
- Consumer Capability – Whether the broker is capable of managing one-to-one and/or one-to-many consumers.
One-to-One
One-to-Many
We checked out the latest and greatest services out there in order to find out which provider is the strongest within these three categories.
Comparing Different Message Brokers
RabbitMQ (AMQP)
Scale: based on configuration and resources, the ballpark here is around 50K msg per second.
Persistency: both persistent and transient messages are supported.
One-to-one vs one-to-many consumers: both.
RabbitMQ was released in 2007 and is one of the first common message brokers to be created. It’s an open source that delivers messages through both point-to-point and pub-sub methods by implementing Advanced Message Queuing Protocols (AMQP). It’s designed to support complex routing logic.
There are some managed services that allow you to use it as a SaaS but it’s not part of the native major cloud provider stack. RabbitMQ supports all major languages, including Python, Java, .NET, PHP, Ruby, JavaScript, Go, Swift, and more.
Expect some performance issues when in persistent mode.
Kafka
Scale: can send up to a millions messages per second.
Persistency: yes.
One-to-one vs one-to-many consumers: only one-to-many (seems strange at first glance, right?!).
Kafka was created by Linkedin in 2011 to handle high throughput, low latency processing. As a distributed streaming platform, Kafka replicates a publish-subscribe service. It provides data persistency and stores streams of records that render it capable of exchanging quality messages.
Kafka has managed SaaS on Azure, AWS, and Confluent. They are all the creators and main contributors of the Kafka project. Kafka supports all major languages, including Python, Java, C/C++, Clojure, .NET, PHP, Ruby, JavaScript, Go, Swift and more.
Redis
Scale: can send up to a million messages per second.
Persistency: basically, no – it’s an in-memory datastore.
One-to-one vs one-to-many consumers: both.
Redis is a bit different from the other message brokers. At its core, Redis is an in-memory data store that can be used as either a high-performance key-value store or as a message broker. Another difference is that Redis has no persistency but rather dumps its memory into a Disk/DB. It’s also perfect for real-time data processing.
Originally, Redis was not one-to-one and one-to-many. However, since Redis 5.0 introduced the pub-sub, capabilities boosted and one-to-many became a real option.
Message Brokers per Use Case
We covered some characteristics of RabbitMQ, Kafka, and Redis. All three are beasts in their category, but as described, they operate quite differently. Here is our recommendation for the right message broker to use according to different use cases.
Short-lived Messages: Redis
Redis’s in-memory database is an almost perfect fit for use-cases with short-lived messages where persistence isn’t required. Because it provides extremely fast service and in-memory capabilities, Redis is the perfect candidate for short retention messages where persistence isn’t so important and you can tolerate some loss. With the release of Redis streams in 5.0, it’s also a candidate for one-to-many use cases, which was definitely needed due to limitations and old pub-sub capabilities.
Large Amounts of Data: Kafka
Kafka is a high throughput distributed queue that’s built for storing a large amount of data for long periods of time. Kafka is ideal for one to many use cases where persistency is required.
Complex Routing: RabbitMQ
RabbitMQ is an older, yet mature broker with a lot of features and capabilities that support complex routing. It will even support complex routing communication when the required rate is not high (more than a few tens of thousands msg/sec).
Consider Your Software Stack
The final consideration, of course, is your current software stack. If you’re looking for a relatively easy integration process and you don’t want to maintain different brokers in a stack, you might be more inclined to work with a broker that is already supported by your stack.
For example, if you’re using Celery for Task Queue in your system on top of RabbitMQ, you’ll have an incentive to work with RabbitMQ or Redis as opposed to Kafka who is not supported and would require some rewriting.
We at Otonomo have used all the above through our platform evolution and growth and then some! It’s important to remember that each tool has its own pro & cons and it’s about understanding them and choosing the right tool for the job and that specific moment, situation and requirements.
Top comments (10)
Nice & comprehensive overview of one of my favourite topics, message oriented middleware (MOM), thank you Assyahid. :-)
Just some remarks from my side:
RabbitMQ is far from being one of the first brokers. Messaging was invented and/or became popular in the 90's with IBM MQ and several JMS implementations. However RabbitMQ might be one of the earliest free message brokers (together with ActiveMQ).
While the 1:1 pattern makes use of queues (where messages are just being queued), I would suggest to explain the 1:n pattern with topics and subscriptions (publish/subscribe). Kafka works that way. RabbitMQ is a bit more complicated, but also doesn't just use queues for 1:n message routing, but introduces exchanges for that matter. I think exchanges are specific to RabbitMQ, at least I haven't seen them anywhere else, so the more popular 1:n implementation is definitely publish/subscribe.
One of the best features of Kafka is its ability to replay messages. It is based on Kafkas append-only log with pointers to the last message that has been fetched from each subscriber. So while message brokers were initially not designed for persistency, Kafka has indeed managed to provide a pretty good persistency. However there's no good way to query data, because Kafka has no idea of the data structures in the messages, so in the end it's still a bad way to persist your data and should only be used additionally (e.g. for streaming) to a database.
very good answer, thank you for providing additional opinions on the topic of discussion of messages broker , I am very happy if someone provides additional opinions because it can open one's view to the development of microservices
I wouldn‘t agree on the last part regarding querying kafka. You can make use of the schema registry and describe the events structure so they can be easily queried (e.g. via ksqldb).
Honestly, if you don't have a conclusion where you probably don't need message brokers, you are really missing a critical aspect. Because even if you are doing something complex, it is still better done without one. Before adding a message broker decide, "can this be done a different way" first. If the answer is yes, then you should absolutely not be using a message broker. If the answer is no, then consult relevant literature.
Telling people to use message brokers in every case is just irresponsible.
Another note about nats.io/. A lighter weight but capable alternative to Kafka. Much easier to run on Kubernetes if required
I think Redis Streams is relevant to this discussion: devopedia.org/redis-streams
Would add nsq.io/ as another interesting alternative.
Why not MQTT?
dev-to-uploads.s3.amazonaws.com/i/...
According to HiveMQ CTO Dominik Obermaier, Kafka can bring lot of value to IoT solutions thanks to its capacity to process large amounts of real-time data providing high throughput and low latency.
Kafka, though, is not by itself especially geared towards IoT applications due to a fundamental mismatch between the requirements of a typical stream-processing solution and those of an IoT solution. In particular, Obermaier mentions the restricted number of topics Kafka consumers and producers may handle in comparison with potentially millions of topics used in an IoT application; the complexity of typical Kafka API implementations which does not lend itself easily to use on constrained devices; the unavailability of key IoT features such as keep alive and last will and testament and others.
This is where the HiveMQ Enterprise Extension for Kafka comes into the picture. One of its key features, indeed, is its ability to map MQTT topics to Kafka topics, thus allowing to overcome Kafka unsuitability to handle large numbers of topics. Similarly, all IoT devices will only see the MQTT broker, so they can keep using MQTT with its reduced requirements and full support for keep alive and last will and testament. Additionally, in cases where the unreliability of the network may break the connection between the broker and the Kafka clusters, the former will buffer all received messages, ensuring they are not lost.
conclusion
Based on a publish/subscribe model, Kafka is one of the most widely used platforms to process and distribute real-time data streams. MQTT is a publish/subscribe protocol particularly suited to IoT applications thanks to its small footprint, real-time guarantees, and suitability for use in high-latency, low-throughput, and unreliable networks.
Might want to consider nats.io which I've been using with my microservices quite heavily recently.