What is RabbitMQ and why do you need a Message Broker?

Asynchronous communication and Message Brokers

In traditional distributed applications, communication was usually done in a synchronous fashion. Synchronous communication, though simple has its own challenges.

Let's take an example of an e-commerce application:

Let's assume our checkout service wants to interact with the inventory service to do a task. The following problems can occur if you are using synchronous communication:

1. There could be a network issue causing the RPC call to fail.
2. The inventory service could be down.
3. What if too many requests come in making the inventory service a bottleneck?

While you can create resiliency mechanisms for synchronous communication like retrying in case of network failures, introducing a circuit breaker to gracefully shut down functions in case the target service is down, spawning additional pods (horizontal scaling) and putting them behind a load balancer to distribute requests in case load on the system increases, there are still things that can go wrong.

This is where asynchronous communication and message queues/brokers come into the picture. They sit in between distributed components.

The checkout service will add an event/message to the message queue and move onto the next task. The inventory service, whenever it's ready (and at it’s own pace) will process these messages.

RabbitMQ

RabbitMQ is a message broker and implements the AMQP (Advanced Messaging Queuing Protocol).

Producers, instead of submitting messages directly to a queue, submit messages to an Exchange. The message can have a Routing Key.

Queues are connected to exchanges using queue Bindings.

Consumers listen to a queue.

An Exchange routes an incoming message to queue(s) using the routing key of the incoming message and the queue bindings. How messages move through the system depends on the type of exchange.

Types of Exchanges

Fanout Exchange

A fanout exchange routes messages to all queues that are bound to it and ignore the routing key.

Topic Exchange

Topic exchanges route messages to queues based on wildcard matches between the routing key and the queue binding.

The routing key must be a list of words, delimited by a period (.). Examples are agreements.us and agreements.eu.stockholm which in this case identifies agreements that are set up for a company with offices in lots of different locations. The routing patterns may contain an asterisk (“”) to match a word in a specific position of the routing key (e.g., a routing pattern of “agreements...b.” only match routing keys where the first word is “agreements” and the fourth word is “b”). A pound symbol (“#”) indicates a match of zero or more words (e.g., a routing pattern of “agreements.eu.berlin.#” matches any routing keys beginning with “agreements.eu.berlin”).

Direct Exchange

With the topic exchange, you can do a partial match between the routing key and queue bindings, while with a direct exchange you can only do a direct/full match.