Choosing a Message Broker: Kafka vs RabbitMQ vs AWS SQS/SNS

Microservice applications rely heavily on messaging and asynchronous communications to keep everything working smoothly.

Choosing the right message broker is one of the first critical choices you must make when developing the services that need to communicate with each other.

Making the “right” choice can be a battle of features and edge cases that can be challenging to differentiate between.

In this article, I’ll help provide a bit of guiding light by providing an overview of a few of the more well-known message brokers – Kafka, RabbitMQ, and AWS SQS/SNS.

I’ll explore the driving forces behind them, the general messaging model they follow and do my best to provide some guidance on selecting the broker that is right for you.

Apache Kafka

Kafka is an open-source message broker developed and maintained primarily by the Apache Software Foundation, with the assistance of the open-source community.

Key Features

• Focus on streamable content, working with large data streams
• Message persistence and reprocessing capabilities are core features
• On-site hosting with third party options

Kafka provides optimized stream-based processing of events, with a publish/subscribe model driving the communications between consumers.

These events can be subdivided into topics, allowing for greater organization of your distributed application’s communication patterns, and are partitioned onto multiple servers within a cluster, allowing for a resilient and highly performant message delivery system.

Technical Details and Deployment

Apache provides SDKs in several different languages.

Kafka is designed to be deployed on-site, within your own application’s architecture. This can be on a set of stand-alone servers, a virtual machine, or a Docker container.

Multiple vendors offer Kafka hosting as a service, such as AWS, CloudKarafka, and Aiven, or in virtual machines.

Below is some sample JS code for getting started with Apache Kafka events:

const { Kafka } = require('kafkajs')
const kafka = new Kafka({
 clientId: 'my-app',
 brokers: ['localhost:9092']
})

// this produces a message
async function produce() {
 const producer = kafka.producer()
 await producer.connect()
 await producer.send({
   topic: 'TOPIC_NAME',
   messages: [
     { key: 'key1', value: 'hello world' },
   ],
 })
}

async function consume() {
 const consumer = kafka.consumer({ groupId: 'my-group' })
 await consumer.connect()
 await consumer.subscribe({ topic: 'TOPIC_NAME' })
 await consumer.run({
   eachMessage: async ({ topic, partition, message }) => {
     console.log({
       key: message.key.toString(),
       value: message.value.toString(),
       headers: message.headers,
     })
   },
 })
}

Strengths & Weaknesses

Kafka has a high focus on data stream throughput, something that shows in their performance statistics.

This focus on processing streams of data results in a system with high throughput, allowing for complex processing of large data streams.

Kafka’s routing capabilities for those streams of data are relatively limited when compared to other message brokers – a gap that is continually getting smaller as these products improve.

To summarize, Kafka is a powerful solution that can provide robust and fault-tolerant high-performance message streaming, letting you confidently drive your application’s behavior.

Depending on your bandwidth and resources, you can abstract away as much or as little of the hosting as you feel comfortable, making Kafka a solid choice that will scale with your traffic.

RabbitMQ

Like Kafka, RabbitMQ is another open-source message broker. Originally developed by Rabbit Technologies, the technology has through a series of acquisitions ended up under the ownership of VMWare.

Key Features

• Focus on messaging-based communication, with support for large data streams
• Complex routing capabilities provided as a core feature
• On-site hosting with third party options

RabbitMQ uses the publish/subscribe model as well, sending message objects in their binary form to different named queues, which can be dynamically created and destroyed.

RabbitMQ is designed to operate both in isolation and as part of a cluster, providing enough configurable power to drive any set of redundancy or data safety needs.

Technical Details and Deployment

RabbitMQ provides several client libraries in a wide variety of languages.

It can be deployed on-site, on anything from a full server to a container, or on one of several cloud providers:

• Stackhero
• CloudAMQP
• Compose

The following sample code, written in Node.js using the AMQPLIB package, should provide a small sample of what it might be like to work with RabbitMQ:

const amqp = require('amqplib/callback_api');
amqp.connect('amqp://localhost', function(error0, connection) {
 if (error0) {
   throw error0;
 }
 connection.createChannel(function(error1, channel) {
   if (error1) {
     throw error1;
   }
   const queue = 'hello-queue';
   const msg = 'Hello world!';

   channel.assertQueue(queue, {
     durable: false
   });

   // Sending message to queue
   channel.sendToQueue(queue, Buffer.from(msg));
   console.log("Sent message", msg);

   // Consuming messages
   channel.consume(queue, function(msg) {
     console.log("Received message", msg.content.toString());
   }, { noAck: true });
 });
});

Strengths & Weaknesses

RabbitMQ has the power to handle workloads of nearly any size, and can effectively scale alongside your application as your user base grows.

With a focus on messaging-based delivery and complex routing scenarios, RabbitMQ is extremely adaptable to any application architecture.

While originally there was no great support for data stream processing, and messages were generally only processed once without the capacity to reprocess a stream, both of these gaps have closed as RabbitMQ has continued to grow.

With the ability to take ownership of the things you want and outsource the rest, RabbitMQ can fit into any appropriate role in your application’s infrastructure.

Amazon Web Services SQS/SNS

SNS and SQS represent two different ways of looking at distributed messaging.

SNS is highly focused on message delivery, offering a publish-subscribe model to quickly distribute a message to an array of clients (e.g., mobile devices, HTTPS endpoints, other AWS services).

SQS, by comparison, is focused on the successful delivery and processing of messages by individual clients.

Key Features

• Two products allowing for both broadcast messaging and pub/sub
• Quick setup and configuration using AWS
• No hosting available outside of AWS

While SNS will broadcast the same message to an array of recipients, SQS will distribute queue messages to single subscribers for processing.

SNS takes a push-based approach to notifications, allowing for automated responses to notification activity, while SQS tends to focus more on a polling-style mechanism with some additional event-driven functionality supported.

Technical Details

AWS provides a general SDK with access to most AWS services (SQS and SNS included) in several popular languages.

The below sample code uses the AWS SDK to demonstrate the process of working with SNS and SQS:

// SNS - publish
const AWS = require('aws-sdk');
AWS.config.update({ region: 'REGION' });

const publishParams = {
 Message: 'MESSAGE_TEXT',
 TopicArn: 'TOPIC_ARN'
};

const publishTextPromise = new AWS.SNS({ apiVersion: '2010-03-31' }).publish(publishParams).promise();

publishTextPromise.then(
 function(data) {
   console.log(`Message ${publishParams.Message} sent to topic ${publishParams.TopicArn}`);
 }).catch(
 function(err) {
   console.error(err, err.stack);
 });

// SNS - Subscribe
const subscribeParams = {
 TopicArn : 'TOPIC_ARN'
}
const subscribePromise = new AWS.SNS({ apiVersion: '2010-03-31' }).listSubscriptionsByTopic(subscribeParams).promise();
subscribePromise.then(
 function(data) {
   console.log(data);
 }).catch(
 function(err) {
   console.error(err, err.stack);
 }
);

// SQS - send
const sqs = new AWS.SQS({ apiVersion: '2012-11-05' });
const queueURL = "SQS_QUEUE_URL";

const sendParams = {
 DelaySeconds: 10,
 MessageAttributes: {
   "Title": {
     DataType: "String",
     StringValue: "Some String"
   }
 },
 MessageBody: "Something!",
 QueueUrl: queueURL
};

sqs.sendMessage(sendParams, function(err, data) {
 if (err) {
   console.log("Error sending to SQS", err);
 } else {
   console.log("Success sending to SQS", data.MessageId);
 }
});

// SQS - receive
const receiveParams = {
 AttributeNames: [
   "SentTimestamp"
 ],
 MaxNumberOfMessages: 10,
 MessageAttributeNames: [
   "All"
 ],
 QueueUrl: queueURL,
 VisibilityTimeout: 20,
 WaitTimeSeconds: 0
};

sqs.receiveMessage(receiveParams, function(err, data) {
 if (err) {
   console.log("Receive Error", err);
 } else if (data.Messages) {
   console.log("Received messages:", JSON.stringify(data.Messages))
 }
});

Strengths & Weaknesses

AWS SQS and SNS, together, can be used to build the backbone of a highly scalable, highly resilient distributed application.

With ties into many other AWS services (such as AWS Lambda), these two tools can help you easily grow your application’s communications, while providing you with all the tools you need to manage the underlying complexities of the service interactions.

If your web application is already running on AWS, set-up time is effectively zero, with much less complexity than many of the other systems. This does potentially come at the expense of a larger AWS bill as the number of messages grows.

While Kafka and RabbitMQ don’t provide a default message size limit, AWS provides some limits around SQS and SNS messages – converting the messages into S3 objects after they reach a certain size.

We published a detailed article on how you can overcome this size limit – I highly recommend browsing it to get a feel for how SQS in particular manages large messages.

[You can find our SQS/SNS Producer/Consumer Library inside. It provides an ability to pass payloads through s3]

SQS and SNS being cloud-first do add the additional complication of being vendor-locked to a specific service, whereas other message brokers resolve this by providing local installation and maintenance capabilities.

Choosing The Right Message Broker

Generally speaking, there are two considerations you should be thinking about when choosing a broker:

Consideration #1: The Type of Messages You Send

The first step to choosing a message broker is determining what messages you’ll be sending, and what their general format will be.

The characteristics of these messages will drive the questions that need to be asked about each platform’s offering, though most will be roughly equivalent in terms of feature set – meaning that at a general level, every solution listed above supports the functionality needed to serve as a message broker for a scalable distributed application.

From a pure feature perspective, one solution is as good as another.

Consideration #2: Your Daily Work and Applications’ Infrastructure

This is where secondary considerations come into play. Think about your day-to-day work and systems and ask yourself:

• Are you building an application exclusively in AWS? Maybe SQS and SNS make the most sense for establishing your inter-service communication
• Are you more interested in writing your application than maintaining the piped data between its components? Then a third-party hosted solution might be the best option for allowing you to focus on your strengths while growing your codebase
• Are you highly focused on speed of delivery and minimal latency? Then Kinesis might be right up your alley (we’ll go over Kinesis on another article, so stay tuned), while an application more focused on verified delivery and redundancy might choose a different technology.

The requirements of your application’s infrastructure and behavioral patterns will drive the choice at this level.

With the above taken into the account, and with the caveat that it is hard – and somewhat unfair – to reduce these large tech products down to a couple of lines of recommendations, here are some guidelines on choosing the right message broker:

• If you care about message retention and being able to easily re-process data, Kafka is likely your best option
• If you are more concerned with being able to maintain and implement a complex set of routing rules, RabbitMQ is likely your best choice
• If you’re a small startup looking to get up and running quickly, with minimal overhead, AWS SQS/SNS is a great option given its quick setup and cost structure

Getting End-to-end Visibility Into a Message Journey

One element you’ll want to evaluate is how to best maintain the final product. Once your application is sending messages, how do we track things down when they go wrong?

OpenTelemetry is a set of SDKs and tools that can be used to set up observability for your distributed application, providing you the means to troubleshoot the distributed messaging driving your application when things go wrong.

Here’s a quick step-by-step guide on implementing OpenTelemetry in your distributed applications, allowing you to achieve end-to-end visibility into a message journey. The guide demonstrates how to do so with Kafka as the message broker.

[PS: grab your OpenTelemetry Kafkajs Instrumentation for Node.js inside.]

Conclusion

If you’re building an application that is “distributed” in any way, odds are you’ll need to handle asynchronous communications between components of your application at some point.

Messages – and the brokers that deliver them – will play a critical role in the infrastructure driving your application.

The above summary is by no means exhaustive – I would probably need another thousand words just to make a solid start at capturing the complete message broker landscape – but should hopefully provide some quality information that you can use when making your decision.

The key is to fully understand the requirements for your application, and how those requirements fit into the capabilities of the message brokers you are evaluating.

There is ultimately no “wrong” answer to which message broker you choose, but hopefully, the above information helps point you in the right direction.

About Aspecto

Aspecto is an OpenTelemetry-based troubleshooting platform 
that helps developers prevent distributed application 
issues from their local dev environment, 
and across the entire development cycle.

You can think of it as the 
Chrome DevTools for your distributed applications.

Aspecto is used for detecting and 
troubleshooting microservices-based distributed systems, 
and preventing software failures before deployment.

Visit us at Aspecto.io
for more microservices tutorials