Hello everyone! Let’s start the hands-on section of the gRPC course. The target of the whole section is to build a "pc book" web service that will allow us to manage and search for laptop configurations.
Here's the link to the full gRPC course playlist on Youtube
Github repository: pcbook-go and pcbook-java
Gitlab repository: pcbook-go and pcbook-java
Protocol buffer basics
In this lecture, we will learn how to write a simple protocol-buffer message with some basic data types, install Visual Studio Code plugins to work with protobuf, and finally we will install protocol-buffer compiler and write a Makefile to run code generation for Go.
But before start, make sure that you already have Go and Visual Studio Code up and running properly on your computer. If not, you can watch my tutorial video on how to install Go and setup Visual Studio code:
The tutorial will guide you, step by step, to install Go, add the bin
folder to your PATH
, install Visual Studio Code, customise its theme and setup Go extensions to work with it.
Once everything is ready, you can come back here and continue this lecture.
Install vscode plugins
Alright, let's start by creating a new project. First, I will create a simple hello-world program in main.go
file and run it, just to make sure that Go is working properly.
package main
import "fmt"
func main() {
fmt.Println("Hello world")
}
Then create a new folder named protoc
, and add a processor_message.proto
file under it.
pcbook
├── proto
│ └── processor_message.proto
└── main.go
Vscode will ask us to install the extensions for the proto file. So let's go to the marketplace and search for ext:proto
There are 2 extensions shown at the top that we should install: clang-format
and vscode-proto3
. Let’s click install for both of them.
How to define a protobuf message
Now come back to our proto file. This file will contain the message definition of the CPU of a laptop.
We start with syntax = "proto3"
.
At the moment, there are 2 versions of protocol buffer on Google's official documentation: proto2
and proto3
. For simplicity, we will only use proto3
(the newer version) in this course.
The syntax is pretty simple, just use the message
keyword followed by the name of the message. Then inside the message block, we define all of its fields as shown in this picture:
Note that the name of the message should be UpperCamelCase, and the name of the field should be lower_snake_case.
There are many built-in scalar-value data types that we can use, for instance: string
, bool
, byte
, float
, double
, and many other integer types. We can also use our own data types, such as enums or other messages.
Each message field should be assigned a unique tag. And the tag is more important than the field name because protobuf will use it to serialise the message.
A tag is simply an arbitrary integer with the smallest value of 1, and the biggest value of 229 - 1, except for numbers from 19000 to 19999, as they're reserved for internal protocol buffers implementation.
Note that tags from 1 to 15 take only 1 byte to encode, while those from 16 to 2047 take 2 bytes. So you should use them wisely, like: saving tags from 1 to 15 for very frequently occurring fields.
And remember that the tags don't need to be in-order (or sequential), but they must be unique for the same-level fields of the message.
Define the CPU message
Now let's get back to our proto file and define the CPU message.
syntax = "proto3";
message CPU {
string brand = 1;
string name = 2;
uint32 number_cores = 3;
uint32 number_threads = 4;
double min_ghz = 5;
double max_ghz = 6;
}
The CPU will have a brand of type string
, such as "Intel", and a name also of type string
, for example "Core i7-9850".
We need to keep track of how many cores or threads the CPU has. They cannot be negative, so let's use uint32
here.
Next, it has the minimum and maximum frequency, for example 2.4 Ghz or something like that. So we can use double
type here.
Generate Go codes
Now we've finished our first protobuf message. How can we generate Go codes from it?
First, we need to install protocol buffer compiler (or protoc
). On macOS, we can easily do that with the help of Homebrew.
You can install Homebrew with this simple command:
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Once Homebrew is installed, can run this command to install protoc
:
brew install protobuf
We can check if it's working or not by running the protoc
command.
Next we will go to grpc.io to copy and run 2 commands to install 2 libraries: the golang grpc
library and the protoc-gen-go
library.
go get -u google.golang.org/grpc
go get -u github.com/golang/protobuf/protoc-gen-go
Now we're all set! I will create a new folder named pb
to store the generated Go codes.
pcbook
├── proto
│ └── processor_message.proto
├── pb
└── main.go
Then run this command to generate the codes:
protoc --proto_path=proto proto/*.proto --go_out=plugins=grpc:pb
Our proto file is located inside the proto
folder, so we tell protoc
to look for it in that folder.
With the go_out
parameter, we tell protoc
to use the grpc
plugins to generate Go codes, and store them inside the pb
folder that we've created before.
Now if we open that folder in vscode, we will see a new file processor_message.pb.go
.
pcbook
├── proto
│ └── processor_message.proto
├── pb
│ └── processor_message.pb.go
└── main.go
Look inside, there's a CPU struct and all fields with the correct data types as we defined in our protocol buffer file.
const _ = proto.ProtoPackageIsVersion3
type CPU struct {
Brand string `protobuf:"bytes,1,opt,name=brand,proto3" json:"brand,omitempty"`
Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"`
NumberCores uint32 `protobuf:"varint,3,opt,name=number_cores,json=numberCores,proto3" json:"number_cores,omitempty"`
NumberThreads uint32 `protobuf:"varint,4,opt,name=number_threads,json=numberThreads,proto3" json:"number_threads,omitempty"`
MinGhz float64 `protobuf:"fixed64,5,opt,name=min_ghz,json=minGhz,proto3" json:"min_ghz,omitempty"`
MaxGhz float64 `protobuf:"fixed64,6,opt,name=max_ghz,json=maxGhz,proto3" json:"max_ghz,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
}
func (m *CPU) Reset() { *m = CPU{} }
func (m *CPU) String() string { return proto.CompactTextString(m) }
func (*CPU) ProtoMessage() {}
func (*CPU) Descriptor() ([]byte, []int) {
return fileDescriptor_466578cecc6db379, []int{0}
}
func (m *CPU) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CPU.Unmarshal(m, b)
}
func (m *CPU) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
return xxx_messageInfo_CPU.Marshal(b, m, deterministic)
}
func (m *CPU) XXX_Merge(src proto.Message) {
xxx_messageInfo_CPU.Merge(m, src)
}
func (m *CPU) XXX_Size() int {
return xxx_messageInfo_CPU.Size(m)
}
func (m *CPU) XXX_DiscardUnknown() {
xxx_messageInfo_CPU.DiscardUnknown(m)
}
var xxx_messageInfo_CPU proto.InternalMessageInfo
func (m *CPU) GetBrand() string {
if m != nil {
return m.Brand
}
return ""
}
func (m *CPU) GetName() string {
if m != nil {
return m.Name
}
return ""
}
func (m *CPU) GetNumberCores() uint32 {
if m != nil {
return m.NumberCores
}
return 0
}
func (m *CPU) GetNumberThreads() uint32 {
if m != nil {
return m.NumberThreads
}
return 0
}
func (m *CPU) GetMinGhz() float64 {
if m != nil {
return m.MinGhz
}
return 0
}
func (m *CPU) GetMaxGhz() float64 {
if m != nil {
return m.MaxGhz
}
return 0
}
There are some special fields used internally by gRPC to serialise the message, but we don't need to care about them. Some useful getter functions are also generated. So it looks great!
Write a Makefile
The command that we used to generate codes is pretty long, so it’s not very convenient to type when we update the proto file and want to regenerate the codes. So let's create a Makefile with a short and simple command to do that.
pcbook
├── proto
│ └── processor_message.proto
├── pb
│ └── processor_message.pb.go
├── main.go
└── Makefile
In this Makefile, we add a gen
task to run code generation command, a clean
task to remove all generated go files whenever we want, and a run
task to run the main.go
file as well.
gen:
protoc --proto_path=proto proto/*.proto --go_out=plugins=grpc:pb
clean:
rm pb/*.go
run:
go run main.go
We can try them in the terminal.
When we run make clean
, the generated files will be deleted.
When we run make gen
, the files will be regenerated in pb
folder.
And finally, when we run make run
, "Hello world" is printed.
What's next
OK, so now you know how to define a simple protocol buffer message and generate Go code from it. In the next lecture, we will dig deeper and learn more advanced features of protobuf.
Thanks for reading! Happy coding, and see you later!
If you like the article, please subscribe to our Youtube channel and follow us on Twitter for more tutorials in the future.
If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.
Top comments (3)
Great course!🤩
The warning
WARNING: Missing 'go_package' option in "processor_message.proto"
generated byprotoc
may be eliminated by adding the lineoption go_package = ".;processor_message";
after
syntax = "proto3";
in processor_message.protoreally save my time! thanks
The command "protoc --proto_path=proto proto/*.proto --go_out=plugins=grpc:pb" is working fine without errors but the files ain't being generated in the "pb" folder
Pls help!!