Why should we use Protobuf in Web API as data transfer protocol.

What is Protobuf?

Protobuf, or Protocol Buffers, is a structured data serialization protocol developed by Google, similar to JSON or XML. Unlike JSON or XML, Protobuf is binary-based, making it not human-readable but highly efficient and fast for server-to-server communication.

Key Features

• Language Neutral and Platform Independent: Protobuf allows you to define your data structure once using a protocol buffer compiler (protoc), and then generate language-specific code for various programming languages such as C#, Java, Go, Node.js (JavaScript or TypeScript), Ruby, Python, and more.
• Efficiency: Due to its binary format, Protobuf is faster and more efficient in serialization and deserialization compared to text-based formats like JSON.
• Widely Used in gRPC: Protobuf is the data serialization format used in gRPC. Google adopts Protobuf for all its service-to-service communications, making it a reliable choice for enterprise adoption.

Example Usage

Protobuf is particularly suitable for applications requiring low latency and high efficiency. Below is a simple example of how to use Protobuf in a C# console application for data serialization and deserialization, and how it compares with JSON.

Defining a Protobuf Message Contract in C

Here, we model a UserInfo message, which is sent to UC to provision Microsoft Teams and add 100 users to it.

syntax = "proto3";
message UserInfo {
  int32 user_id = 1;
  string user_name = 2;
  string email = 3;
}

In the above Protobuf message contract, all attributes are self-descriptive. Each attribute is assigned a unique number, which is used during serialization and deserialization, not the attribute name. This helps maintain a small message size and ensures forward and backward compatibility.

Note: Clients and services will ignore field numbers they do not recognize. For more details about Protobuf, visit protobuf.dev.

Using Protobuf in C# with Protobuf-net

You can use the Protobuf compiler to generate code for serialization and deserialization, or you can use the .NET NuGet package Protobuf-net to write a C# style class as a Protobuf contract.

To get started, download the Protobuf-net NuGet package from NuGet.

Here is an example of how to define a Protobuf message in C#:

[ProtoContract]
public class UserInfo {
  [ProtoMember(1)]
  public int UserId { get; set; }

  [ProtoMember(2)]
  public string UserName { get; set; }

  [ProtoMember(3)]
  public string Email { get; set; }
}

Here is my C# code which does both JSON and protobuf data Serialization and Deserialization and save the result to file.
// See https://aka.ms/new-console-template for more information
using System.Text.Json;
using ProtoBuf;

List users = new List();

for (int i = 0; i < 100; i++)
;

users.Add(user);

}

// JSON Seralizer
string jsonString = JsonSerializer.Serialize(users);

using (var file = File.Create("users.json"))

using (var file = File.Create("users.data"))

List users;
using (var file = File.OpenRead("users.data"))

See the file size of .json and users.data . JSON format is 9.8k and Protobuf format size is 6.7k

Performance Comparison: JSON vs. Protobuf

Protobuf is significantly more efficient than JSON in terms of serialization and deserialization speed and the size of the data transferred over the wire. Here's a percentage difference for 100 users' data:

• JSON: Larger size, slower serialization/deserialization.
• Protobuf: Smaller size, faster serialization/deserialization.

When to Use Protobuf

1. Microservices: Ideal for service-to-service communication.
2. Language Interoperability: When the application involves multiple programming languages.
3. Performance: When fast and efficient serialization/deserialization is required.
4. Compatibility: When backward and forward compatibility between client and server contracts is necessary.
5. Storage Efficiency: For saving data to disk with less storage and performing ETL operations.
6. Large Payloads in APIs: For REST API routes with large payloads, where JSON is slow and cumbersome over the wire.

Note: If you want to use the Google Protobuf compiler for code generation, download it from GitHub based on your OS. Follow this link for code generation instructions.

For more information, visit Protobuf Documentation.

Thanks,

Sreeni Ramadurai