The Simple Guide to Message Queues

Congratulations, your officially a junior developer.

One day you hear your colleagues talking about tech you've never heard of, things such as Docker, Kubernetes, RabbitMQ, Kafka, etc.

Then imposter syndrome hits, and you feel overwhelmed.

Don't worry you're not alone

No need to be discouraged, most if not all developers felt this way at some point.

In reality, it's a lot simpler than you think - below we will talk about one set of technologies that you probably heard but never really understood.

Table of Contents

1. What is a message queue?
2. Analogy
3. What is the AMQP? (Advanced Message Queue Protocol)
4. What problems does it fix?
5. Benefits
6. An inside look
7. Exchange Types
8. Message Queue Software
9. Conclusion
10. References

What is a message queue?

If you google this exact question, your gonna get something like this:

In computer science, message queues are software-engineering components typically used for inter-process communication, or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content.

I know this sounds so damn complicated but trust me it isn't.

A message queue is simply a queue of messages.

No shit Sherlock.

I'm sorry if you expected some magical definition, it actually is just a queue of messages. To go further into detail let us define what are queues and messages.

• Message: Data transported between the sender and the receiver application. It's basically an array of bytes with some meta-data. An example of a message could be something that tells one system to start a task, or it could just be a plain text message.
• Queue: a line of things waiting to be handled, starting at the beginning of the line and processing it in sequential order.

With this knowledge in hand, we can create a new and updated definition for message queues:

A message queue is a software component that provides an asynchronous communications protocol, which puts messages in a queue and does not need an immediate response from the receiver.

Analogy

Imagine one day you're hungry and you craving some pizza. Also, imagine it's pre-covid which means you can go to your favorite pizza restaurant which should be Uncle T's pizza. You go in, you order some pizza, you pay for pizza, then the cashier gives you a check and tells you to wait till your pizza is done. After some time that cashier calls you up, gives you your pizza. You go home after that and happily eat your pizza.

You're probably thinking how the hell is this related to message queues?

Well, message queues are essentially the same thing, when a client makes a request to a service, the service doesn't force the client to wait till the request is processed. It gives it back a confirmation that the request is sent to the queue and will soon be processed.

What is AMQP?

The same way how the web works mainly with HTTP, message queues have their own protocol called Advanced Message Queue Protocol aka AMQP.

The official definition is:

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware.

Some features that AMQP provides:

1. Message orientation
2. Queuing
3. Routing (including point-to-point and publish-subscribe)
4. Reliability and security

What problem does it fix?

Imagine you have two services communicating with each other. Let's call them Service A and Service B.

Let's go through the process:

1. Service A sends a request to Service B
2. Service B receives the request and processes the computation
3. Service B returns the response to Service A
4. Service A gets the response and closes the connection.

The problem with this approach is that while Service B is processing the computation, Service A is just stuck waiting, in our analogy above we wouldn't want our customer to be waiting in-front of the cashier till his pizza is ready, he would block other customers from making their order. Same thing here, Service A is simply waiting, not doing anything, and blocking other services from making requests to Service B.

A better approach would be to use a message queue:

Let's go through the steps once again but this time with a message queue system:

1. Service A sends a request to the message queue.
2. Message queue receives the request and adds it to it's queue.
3. Message queue sends back a response, saying that it has added the request and will be processed shortly.
4. Service A closes the connection happily.
5. Once service B is ready, it receives the request from the message queue.
6. Service B process the request successfully
7. Service B sends back a response to the message queue saying it has successfully processed the request.
8. The message queue pops the request out of the queue.

You can even scale this:

Benefits

Scaling: As you can see in the above example we can easily scale this to support many different producers (services that make a request to the message queue) and consumers (services that consume the request).

Redundancy: After storing the message, the queue assures the message will only be removed once the process that reads it confirms its success in reading and processing. If anything goes wrong, the message won't be lost and reprocessed later.

Resiliency: Let's say, for example, that your system consists of two microservices, one for processing orders and another for sending emails. Having a queue to indicate that an email needs to be sent means that even if your email system is down, the order-processing microservice can keep receiving and processing orders. When the email service is back online, it can start reading the messages and sending the emails.

Asynchronous Messaging: In cases where your application doesn't require a response right way for a process, the service that consumes these messages can run its business logic at its own pace.

An inside look

Now let's try to understand how it works under the hood.

Let us break this down:

Producer (also called Publisher): is the client that produces the request, it can be a website, mobile app, microservice, etc.

Exchange: is the place where your message gets received, and then gets routed to the appropriate queue.

Queue: is the place where your message waits till it's ready to be served to the consumer.

Consumer (also called Subscriber): receives and processes messages from the queue.

Exchange Types

As we learned above messages get distributed in the exchange, they usually use some sort of algorithm to decide which message goes to which queue. There are in fact multiple ways/algorithms to distribute messages. These include:

Fanout Exchange means that the message gets copied and goes to all available queues.

Direct Exchange means that it sends the message to the designated queue by checking if the routing key is equal to the binding key.

Topic Exchange allows partial matching of keys. Basically what it means is that you got a message with a routing key red. Then the message can be sent to queues such as red.green or red.blue, etc.

Header exchange basically is like the direct exchange but instead of comparing the routing key to the binding key, it compares the message header with the binding key.

Default (Nameless Exchange) is only unique to RabbitMQ. This exchange compares the routing key with the queue name. When the routing key matches with the name of the queue. It forwards the message to that queue.

Message Queue Software

Now that you've learned the basics of message queues, you can finally try experimenting with them. You don't have to create your own from scratch, there are many open-source software that can fill your needs such as:

RabbitMQ is the most popular open-source message broker, with more than 35,000 production deployments worldwide. RabbitMQ is lightweight and easy to deploy on-premises and in the cloud and runs on all major operating systems.

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Apache ActiveMQ is the most popular open-source, multi-protocol, Java-based messaging server. It supports industry-standard protocols so users get the benefits of client choices across a broad range of languages and platforms.

Conclusion

By the end of this, I hope you at least have a basic idea of what a message queue is. I know this topic might be a bit confusing but play around with it, and I'm sure you will get it.

PS. I always appreciate criticism, so if you got any leave them down in the comments.

Additional References:

• https://lostechies.com/derekgreer/2012/03/28/rabbitmq-for-windows-exchange-types/
• https://www.youtube.com/watch?v=deG25y_r6OY
• https://www.youtube.com/watch?v=7rkeORD4jSw
• https://dzone.com/articles/exploring-message-brokers
• https://www.youtube.com/watch?v=Cie5v59mrTg
• https://www.youtube.com/watch?v=oq1fOr6Ryws
• https://www.youtube.com/watch?v=hfUIWe1tK8E
• https://www.youtube.com/watch?v=O1PgqUqZKTA&list=PLQnljOFTspQVcumYRWE2w9kVxxIXy_AMo&index=4