How to use Protobuf with Go?

Have you ever been in a situation where you do not know the structure of data that services are either consuming or publishing?

Did you know that you have to write a schema only once which defines the structure data and the same schema can be used as validation before sending & receiving data using Protocol Buffers?

In this tutorial, you will learn about using Protobuf. You will send & receive a message asynchronously using Protobuf with RabbitMQ.

Table Of Contents

• Prerequisites
• Protobuf
• Bootstrapping Project
• Creating Protobuf Messages
• Publishing a Message
• Consuming a Message
• Conclusion

Prerequisites

This tutorial will be a hands-on demonstration. If you’d like to follow along, be sure you have the following:

• You have running Docker Composer in your machine.
• You have a basic understanding of Docker, GO, Rabbitmq.

Protobuf

Protobufs are the Messaging Contracts. They are schema that describes the structure of the message. These files are written in .proto extension which is language-neutral i.e write once & use anywhere. Proto Messages are the structured schema that is inside .protofiles

You will write Proto Messages and with the compiler called Proto Compiler, you can generate the code which will have a way to serialize & deserialize data into that format.

There are many benefits of using Messaging Contracts like Protobuf :

1. With these Proto Messages, you will know what type of data you are sending & receiving
2. Serves as a documentation
3. Supports validation for serializing & deserializing
4. Proto messages are language-neutral and it supports various programming languages which means you can publish messages from Go & start consuming them from Php / Node etc

Bootstrapping Project

For creating a new test project follow the steps below:

• Copy Dockerfile & docker-compose.yaml from this repository which will dockerized Go source code, Proto compiler & RabbitMQ
• Create a file publish.go where you will write some code to publish events
• Create a file consume.go where you will write some code to consume events

Creating Protobuf Messages

In an application, while you sending messages you will serialize data using the definition from Proto Messages. And when consuming messages you will deserialize them using the same Proto Messages. Let's create a Proto Message so your services can use them to send and receive messages.

1. Create a folder messages and inside that folder create a file called user_registered.go
2. Inside user_registered.go copy the following codes. Let's understand the code

• syntax defines which proto version you are going to use
• go_package defines the path where all your generated Protobuf code will be into
• `import “validate/validate.proto”; it is importing the validator so that you can validate message using various rules without writing it by yourself
• All message starts with a message keyword followed by name of the message. Inside the block, you can write key-value pairs where the key defines name & type & the value defines its order & rules if there are any. In our case, you are defining the first field userId which is an integer and should be greater than 0 and the next is an email which is a string and should not be empty.

\3. Run make build-proto on your project root. This command will generate two files user_registered.pb.go and user_registered.pb.validate.go that define type & includes all the validation logic that you specified on the text which you can later use on your code while serializing & deserializing.

Publishing a Message

As you have successfully created a Proto Message. Let's use the same auto-generated code while sending messages to other services for this you need to write a way to publish messages. Follow the following steps:

1. Create a directory pkg and a file rabbitmq.go inside pkg dir where you will make a connection with RabbitMQ. The following code the function getConnection is importing library and making a connection with RabbitMQ server and upon error function failOnError is terminating the application with an error.

\2. After successfully making a connection, write a function that will publish a message to RabbitMQ. The following code accepts queueName and payload as a message which will write a message to default exchange to JSON format.

\3. Create publish.go which will import publish function from rabbitmq.go to send data. The following code is using a struct that was generated by the proto using the proto compiler and it is encoding them into JSON after that it is calling to publish the function that you created earlier to send data to UserRegisteredqueue.

Try running make publish to verify that your code is actually working if you see Data has been published the output that means you were successful in publishing a message.

Consuming a Message

Until now you defined a photo message then you created a rabbitmq.goto create a way to publish messages & you used the same function publish.go to publish messages to RabbitMq. Now let's consume the message that you published earlier.

1. Inside rabbit. go add a function that will help in consuming a message. The following code makes a connection to RabbitMq & returns a channel that you can leverage to listen to incoming messages.

\2. Inside consume.go listen to all the messages and then print them. Let's understand the following code :

• The main function is listening to message by creating a consuming client. While deserializing message it is importing UserRegistered struct from the auto-generated code to define the type of data. There might be cases where the producer might send invalid data due to various reasons so it is better to validate that data you are consuming are as per the contracts that you specified.
• validateMessage accepts the JSON data and tries to validate as per the user_registered.proto definition and if the data breaks any validation rules it will throw a validation error

Conclusion

Throughout this tutorial, you have learned Protobuf & how to use them with GO & RabbitMQ.

Protobuf is one of the ways to implement Messaging Contracts there are more options like Pact, AsyncApi

Now with this knowledge, you can use Protobuf on any programming language to send & receive the data. You have also learned that it was easier to predict the data that you are consuming or publishing.

Let me know if you have any comments /suggestions/feedback.

Source code: https://github.com/ujwaldhakal/golang-protobuf
Ref: https://stackshare.io/protobuf