Top 6 message queues for distributed architectures

Message queues for decoupled distributed architectures and their advantages and disadvantages

⚡ Tl;dr

• Message queues can be used for asynchronous communication between services and for processing batched workloads.
• As applications become decoupled, they often need mechanisms to share state, mutate data and handle events in different areas of the system.

🚀 Let’s kick-off

As architectures move toward the cloud and become increasingly distributed, the need for communication and messaging is becoming a critical part of modern software systems. As applications are decoupled into independent building blocks, they often need mechanisms to share state, mutate data and handle events in different areas of the system.

❔ What is a message queue?

Message queues can be used for asynchronous communication between two or more services. They’re great for processing large or batched workloads and help smooth out their processing. Messages are usually pushed into the queue and processed sequentially by the receiver.

Let’s explore both open-source and cloud-hosted message queues.

1️⃣ RabbitMQ

RabbitMQ is an open-source message broker for asynchronous messaging, queueing and delivery acknowledgement. It can be deployed in the cloud or on-premise with Kubernetes, Chef, Docker or Puppet.

👍 Advantages

• Supports a wide range of messaging patterns and easily scales horizontally by adding more nodes to the cluster.
• RabbitMQ has many official and unofficial client libraries and developer tools linked on its website.
• Provides durability for its message by storing backups on disk through other nodes in the cluster.

👎 Disadvantages

• Can be slow at processing larger datasets, typically handling tens of thousands of events per second.
• Some integration services, such as intra-cluster compression and warm standby replication, are only available as part of their paid commercial offering.
• RabbitMQ is somewhat complex and will require technical expertise to set up and maintain over time.

2️⃣ Amazon Simple Queue System (SQS)

Amazon Simple Queue System or SQS is a fully managed message queue built for distributed and serverless platforms. Out of the box, it allows you to send, store and receive any volume of messages between services. It has a free and paid tier.

👍 Advantages

• Simple to set up and requires minimal maintenance over time as SQS is a managed service by Amazon.
• Automatically handles high availability and redundancy by storing messages across multiple availability zones in the region.
• Highly scalable in a way that’s automatically handled based on the number of messages being processed.

👎 Disadvantages

• It may be difficult to estimate the cost of using SQS as you’re charged for each message based on its size and network bandwidth used.
• Latency is variable based on the number of messages being sent and received but is generally less than that of self-hosted solutions.
• The maximum message size is only 256KB.
• SQS doesn’t support real-time data streaming or parallel processing of messages.

3️⃣ Apache ActiveMQ

Apache ActiveMQ is an open-source Java-based message queue that can be accessed by clients written in Javascript, C, C++, Python and .NET. There are two versions of ActiveMQ, the existing “classic” version and the next generation “Artemis” version, which is currently being worked on.

👍 Advantages

• Lots of flexibility for customizing options such as prefetch size, threading, notification to producers and failover.
• You can configure ActiveMQ to guarantee message delivery to ensure they are delivered at least once.

👎 Disadvantages

• It requires manual installation, configuration and maintenance, meaning you’ll likely need someone technical to manage it.
• The ActiveMQ documentation is difficult to read, understand and find the information you’re looking for.

4️⃣ Apache Kafka

Apache Kafka is an open-source, distributed event streaming platform with message communication and storage capabilities. Although Kafka is not technically a message queue, it has the functionality of a message queue using a topic partition.

👍 Advantages

• Easily scales horizontally and span multiple data center and cloud environments.
• Kafka is reliable, with high availability and fault tolerance through replication and partitioning.
• Kafka now has libraries for most languages, including Java, C++, Python, Go, .NET, Rub and NodeJS.

👎 Disadvantages

• Kafka has a steep learning curve due to its technical complexity, likely requiring in-house or external experts.
• Kafka doesn’t have an official interface for management and monitoring, and third-party tools may be required.

5️⃣ Google Cloud Pub/Sub

Google Cloud Pub/Sub is a fully-managed, globally scalable and secure queue provided by Google Cloud for asynchronous processing messages. Cloud Pub/Sub has many of the same advantages and disadvantages as SQS due to also being cloud hosted. It has a free and paid tier.

👍 Advantages

• High scalability and availability that requires no setup and is managed for you automatically by Google.
• The maximum message payload size is 10MB, considerably higher than most other solutions.
• Supports push and pull delivery types.
• The easy-to-use interface on the Google Cloud console for managing and debugging topics and subscriptions.

👎 Disadvantages

• It can be difficult to calculate how much you will be billed for Pub/Sub, as they charge for throughput, storage and egress as described on their length and complex pricing page.
• Low latency for message delivery is not guaranteed as Google prioritizes scalability over speed for Pub/Sub.

6️⃣ Microsoft Azure Service Bus

Microsoft Azure Service Bus is a reliable, fully managed Cloud service for delivering messages via queues or topics. It has a free and paid tier.

👍 Advantages

• High reliability and scalability are automatically provided as this service is fully managed by Microsoft Azure.
• Azure Service Bus integrates well with other products in the Azure ecosystem.

👎 Disadvantages

• The learning curve is high for new users getting started with Azure Service Bus as it can be complex to understand due to poor documentation.
• Azure Service Bus has a maximum message size of 256KB.

🏁 To wrap up

Let us know if you’ve had your own experiences with any of these message queues and comment with your own experiences, pros and cons.

Stay chill 🧊