Introduction to Protocol Buffers

Background

In short, protobuf (Protocol Buffers) is a mature open source project from Google for serializing structured data.

Serialization is a powerful tool that can be used for a variety of purposes in programming. By understanding how it works, we can write more efficient and robust applications.

Serialization can be implemented in a variety of ways, but the general idea is to convert the data into a format that can be easily stored or transmitted. Perhaps the most common data-interchange format today is JSON.

The concept of serializing data is used everywhere, any time you need to communicate (move data) you probably need to do some serialization of data. It enables features such as persistence of data in file, data in HTTP requests, or interoperability between different applications written in different programming languages.

So why Protobuf? Isn't JSON enough?

There are couple of reason why you would want to use Protobuf for your application.

• Do you need to store or transfer much data? This format is not text based as JSON, instead it is in a binary format which will make it more compact.
• Types, increases the maintainability of the code.
• It is extensible, it is pretty easy to add new fields to an already existing data structures, which makes it possible to evolve the format when requirements are changing.
• Well used, there is support for the format in many programming languages.
• Integrating with a system that is already using Protobuf as a communication format.

Of course, there are plenty of other alternatives to Protobuf like JSON, Apache Avro, XML, Apache Thrift to mention some popular ones. They all have their own pros and cons just as Protobuf does.

The interface definition language, IDL

The current version of protocol buffers is version 3, aka proto3.

The description of the data structures are written in a file called .proto. Which is later compiled/transpiled to your language of choice, C#, Java, Python or whatever you want.

There are several named elements that can be used, however the, message, element might be one of the most used ones.

// Specify which version of the proto syntax is used
syntax = "proto3";

// Package declaration, name-spacing
package example;

// We could also import other proto files here.
// import "src/somefile.proto";

// Enumeration definition
enum Role {
  ADMIN = 0:
  VIEWER = 1;
}

// Message definition
message User {
  int32 id = 1;
  string name = 2;
  Role role = 3;
  optional int32 age = 4;
}

The above code should give you a hint on how a proto file looks like. For more details, see the language guide.

When the proto code is written the next thing is to generate the actual code for your language. This is done by compiling the .proto files which will then generate the code needed to serialize and de-serialize the defined messages in the file.

I hope this blog post has given you a good introduction of proto files. If you have any questions, please leave a comment below.

In the next post, I'll show you how to use proto files to create a working example. Stay tuned!