DEV Community

Cover image for πŸ“ˆ Working with RabbitMQ in Golang by examples
Vic ShΓ³stak
Vic ShΓ³stak

Posted on • Updated on

πŸ“ˆ Working with RabbitMQ in Golang by examples

Introduction

Hey, DEV people! πŸ˜‰ Today, I'll cover the topic of working with a Message broker called RabbitMQ in your Go projects. There will be both a theoretical part and practice.

Of course, the article is more aimed at those who just want to understand this topic. But I call on more experienced people to help cover this topic even better in the comments on this article. Together we can do a lot!

πŸ“ Table of contents

What is a message broker?

This is an architectural pattern in distributed systems, where a message broker is an application that converts a single protocol message from the source application to a protocol message from the destination application, thereby acting as an intermediary between them.

Also, the tasks of the message broker include:

  • Checking the message for errors;
  • Routing to specific receiver(s);
  • Splitting the message into several smaller ones, and then aggregating the receivers' responses and sending the result to the source;
  • Saving the messages to a database;
  • Calling web services;
  • Distributing messages to subscribers;

πŸ€” But what is it anyway? Well, let's translate it into our language.

If you simplify this huge description, you can portray the message broker as a post office in real life (which you have encountered many times):

  1. A sender (user of your product) brings a parcel (any data) to the post office and specifies the addressee for receipt (another service).
  2. A post office employee accepts the parcel and places it in the storage area (puts it in the queue to be sent) and issues a receipt that the parcel has been successfully accepted from the sender.
  3. After some time, the parcel is delivered to the addressee (another service), and he doesn't have to be at home to accept the parcel. In this case, his parcel will wait in a mailbox until he receives it.

message broker schema

↑ Table of contents

What problems will be able to solve?

One of the most important problems that can be solved by using this architectural pattern is to parallelize tasks with a guaranteed result, even if the receiving service is unavailable at the time of sending the data.

With the total dominance of microservice architecture in most modern projects, this approach can maximize the performance and resilience of the entire system.

πŸ‘Œ It sounds a bit confusing... but let's use the post office analogy again!

Once the sender gives his parcel to the post office employee, he no longer cares how his parcel will be delivered, but he does know that it will be delivered anyway!

↑ Table of contents

Full project code

For those who want to see the project in action:

GitHub logo koddr / tutorial-go-fiber-rabbitmq

πŸ“– Tutorial: Working with RabbitMQ in Golang by examples.

↑ Table of contents

Setting up RabbitMQ

As usual, let's create a new Docker Compose file:

# ./docker-compose.yml

version: "3.9"

services:

  # Create service with RabbitMQ.
  message-broker:
    image: rabbitmq:3-management-alpine
    container_name: message-broker
    ports:
      - 5672:5672   # for sender and consumer connections
      - 15672:15672 # for serve RabbitMQ GUI
    volumes:
      - ${HOME}/dev-rabbitmq/data/:/var/lib/rabbitmq
      - ${HOME}/dev-rabbitmq/log/:/var/log/rabbitmq
    restart: always
    networks:
      - dev-network

networks:
  # Create a new Docker network.
  dev-network:
    driver: bridge
Enter fullscreen mode Exit fullscreen mode

☝️ Please note! For the initial introduction to RabbitMQ we will not create a cluster and use a load balancer. If you want to know about it, write a comment below.

↑ Table of contents

Setting up Fiber as a sender

To connect to the message broker, we will use the Advanced Message Queuing Protocol or AMQP for short. The standard port for RabbitMQ is 5672.

Okay, let's write a simple data sender using Fiber web framework:

// ./sender/main.go

package main

import (
    "log"
    "os"

    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/logger"
    "github.com/streadway/amqp"
)

func main() {
    // Define RabbitMQ server URL.
    amqpServerURL := os.Getenv("AMQP_SERVER_URL")

    // Create a new RabbitMQ connection.
    connectRabbitMQ, err := amqp.Dial(amqpServerURL)
    if err != nil {
        panic(err)
    }
    defer connectRabbitMQ.Close()

    // Let's start by opening a channel to our RabbitMQ
    // instance over the connection we have already
    // established.
    channelRabbitMQ, err := connectRabbitMQ.Channel()
    if err != nil {
        panic(err)
    }
    defer channelRabbitMQ.Close()

    // With the instance and declare Queues that we can
    // publish and subscribe to.
    _, err = channelRabbitMQ.QueueDeclare(
        "QueueService1", // queue name
        true,            // durable
        false,           // auto delete
        false,           // exclusive
        false,           // no wait
        nil,             // arguments
    )
    if err != nil {
        panic(err)
    }

    // Create a new Fiber instance.
    app := fiber.New()

    // Add middleware.
    app.Use(
        logger.New(), // add simple logger
    )

    // Add route.
    app.Get("/send", func(c *fiber.Ctx) error {
        // Create a message to publish.
        message := amqp.Publishing{
            ContentType: "text/plain",
            Body:        []byte(c.Query("msg")),
        }

        // Attempt to publish a message to the queue.
        if err := channelRabbitMQ.Publish(
            "",              // exchange
            "QueueService1", // queue name
            false,           // mandatory
            false,           // immediate
            message,         // message to publish
        ); err != nil {
            return err
        }

        return nil
    })

    // Start Fiber API server.
    log.Fatal(app.Listen(":3000"))
}
Enter fullscreen mode Exit fullscreen mode

As you can see, at the beginning we create a new connection to RabbitMQ and a channel to send data to the queue, called QueueService1. With a GET request to localhost:3000/send, we can pass a needed text in a msg query parameter, which will be sent to the queue and next to the consumer.

Now create a new Dockerfile called Dockerfile-sender in the root of the project in which we describe the process of creating the container for sender:

# ./Dockerfile-sender

FROM golang:1.16-alpine AS builder

# Move to working directory (/build).
WORKDIR /build

# Copy and download dependency using go mod.
COPY go.mod go.sum ./
RUN go mod download

# Copy the code into the container.
COPY ./sender/main.go .

# Set necessary environment variables needed 
# for our image and build the sender.
ENV CGO_ENABLED=0 GOOS=linux GOARCH=amd64
RUN go build -ldflags="-s -w" -o sender .

FROM scratch

# Copy binary and config files from /build 
# to root folder of scratch container.
COPY --from=builder ["/build/sender", "/"]

# Command to run when starting the container.
ENTRYPOINT ["/sender"]
Enter fullscreen mode Exit fullscreen mode

All that remains is to update the Docker Compose file so that it takes this Dockerfile into account when creating a container with RabbitMQ:

# ./docker-compose.yml

version: "3.9"

services:

  # ...

  # Create service with Fiber sender.
  sender:
    container_name: sender
    ports:
      - 3000:3000
    build:
      context: .
      dockerfile: Dockerfile-sender
    environment:
      AMQP_SERVER_URL: amqp://guest:guest@message-broker:5672/
    restart: always
    networks:
      - dev-network
    depends_on:
      - message-broker

  # ...

# ...
Enter fullscreen mode Exit fullscreen mode

Setting up a message consumer

The message consumer should be able to accept messages from the broker's queue and output in the logs the message sent from the sender.

Let's implement such a consumer:

// ./consumer/main.go

package main

import (
    "log"
    "os"

    "github.com/streadway/amqp"
)

func main() {
    // Define RabbitMQ server URL.
    amqpServerURL := os.Getenv("AMQP_SERVER_URL")

    // Create a new RabbitMQ connection.
    connectRabbitMQ, err := amqp.Dial(amqpServerURL)
    if err != nil {
        panic(err)
    }
    defer connectRabbitMQ.Close()

    // Opening a channel to our RabbitMQ instance over
    // the connection we have already established.
    channelRabbitMQ, err := connectRabbitMQ.Channel()
    if err != nil {
        panic(err)
    }
    defer channelRabbitMQ.Close()

    // Subscribing to QueueService1 for getting messages.
    messages, err := channelRabbitMQ.Consume(
        "QueueService1", // queue name
        "",              // consumer
        true,            // auto-ack
        false,           // exclusive
        false,           // no local
        false,           // no wait
        nil,             // arguments
    )
    if err != nil {
        log.Println(err)
    }

    // Build a welcome message.
    log.Println("Successfully connected to RabbitMQ")
    log.Println("Waiting for messages")

    // Make a channel to receive messages into infinite loop.
    forever := make(chan bool)

    go func() {
        for message := range messages {
            // For example, show received message in a console.
            log.Printf(" > Received message: %s\n", message.Body)
        }
    }()

    <-forever
}
Enter fullscreen mode Exit fullscreen mode

Okay, in the same way with sender, let's create a new Dockerfile called Dockerfile-consumer to describe the process of creating a container for the message consumer:

# ./Dockerfile-consumer

FROM golang:1.16-alpine AS builder

# Move to working directory (/build).
WORKDIR /build

# Copy and download dependency using go mod.
COPY go.mod go.sum ./
RUN go mod download

# Copy the code into the container.
COPY ./consumer/main.go .

# Set necessary environment variables needed 
# for our image and build the consumer.
ENV CGO_ENABLED=0 GOOS=linux GOARCH=amd64
RUN go build -ldflags="-s -w" -o consumer .

FROM scratch

# Copy binary and config files from /build 
# to root folder of scratch container.
COPY --from=builder ["/build/consumer", "/"]

# Command to run when starting the container.
ENTRYPOINT ["/consumer"]
Enter fullscreen mode Exit fullscreen mode

Yes, you guessed it! πŸ˜… Once again, we will have to put a description in the Docker Compose file to create the container for the consumer:

# ./docker-compose.yml

version: "3.9"

services:

  # ...

  # ...

  # Create service with message consumer.
  consumer:
    container_name: consumer
    build:
      context: .
      dockerfile: Dockerfile-consumer
    environment:
      AMQP_SERVER_URL: amqp://guest:guest@message-broker:5672/
    restart: always
    networks:
      - dev-network
    depends_on:
      - sender
      - message-broker

# ...
Enter fullscreen mode Exit fullscreen mode

Great! Finally, we're ready to put everything together and run the project.

↑ Table of contents

running the project

Running the project

Just run this Docker Compose command:

docker-compose up --build
Enter fullscreen mode Exit fullscreen mode

Wait about 1-2 minutes and make a few HTTP request to the API endpoint with different text in msg query:

curl \
    --request GET \
    --url 'http://localhost:3000/send?msg=test'
Enter fullscreen mode Exit fullscreen mode

Next, go to this address: http://localhost:15672. Enter guest both as login and password. You should see the RabbitMQ user interface, like this:

rabbitmq gui

To simply view the logs and metrics inside the Docker containers, I recommend to use a command line utility, called ctop:

ctop

If you turn off the consumer container, but continue to make HTTP requests, you will see that the queue starts to accumulate messages. But as soon as the consumer is started up again, the queue will clear out because the consumer will get all the messages sent.

🎊 Congratulations, you have fully configured the message broker, sender, consumer and wrapped it all in isolated Docker containers!

↑ Table of contents

Photos and videos by

P.S.

If you want more articles (like this) on this blog, then post a comment below and subscribe to me. Thanks! 😻

And of course, you can help me make developers' lives even better! Just connect to one of my projects as a contributor. It's easy!

My projects that need your help (and stars) πŸ‘‡

  • πŸ”₯ gowebly: A next-generation CLI tool for easily build amazing web applications with Go on the backend, using htmx & hyperscript and the most popular atomic/utility-first CSS frameworks on the frontend.
  • ✨ create-go-app: Create a new production-ready project with Go backend, frontend and deploy automation by running one CLI command.
  • πŸƒ yatr: Yet Another Task Runner allows you to organize and automate your routine operations that you normally do in Makefile (or else) for each project.
  • πŸ“š gosl: The Go Snippet Library provides snippets collection for working with routine operations in your Go programs with a super user-friendly API and the most efficient performance.
  • πŸ„β€β™‚οΈ csv2api: The parser reads the CSV file with the raw data, filters the records, identifies fields to be changed, and sends a request to update the data to the specified endpoint of your REST API.
  • 🚴 json2csv: The parser can read given folder with JSON files, filtering and qualifying input data with intent & stop words dictionaries and save results to CSV files by given chunk size.

Top comments (2)

Collapse
 
wagslane profile image
Lane Wagner

I've been working on github.com/wagslane/go-rabbitmq for a few months now, would love it you checked it out!

Collapse
 
ingfdoaguirre profile image
Fernando Aguirre • Edited

Hi Im getting this error randomly on the sender container

UNEXPECTED_FRAME - expected content body, got non content body frame instead

Do you know hos to solve this? thank you

P.S. I had PHP 8 and Symfony 5 running a message sender, I had a 10ms response time, now I have 1ms with GO.

dev-to-uploads.s3.amazonaws.com/up...