Simple CLI To-Do application with Twirp RPC

#go #rpc #twirp #todo

Introduction

You've seen it on Twitter, you've read about it on Medium, it's in all your newsletters... so what is RPC? In this post I'll introduce you to the technology using TwitchTv's fantastic RPC library (Twirp) and quench your curisosity. We'll be building a classic tutorial application... the To-Do list, but this time, making use of a CLI for interaction. Do you even know how to use a framework if you don't build a To-Do list with it?! Jokes aside, a To-Do list is the perfect size and requires simple CRUD utility to work, making it a great way to dip your feet into something new.

Why RPC?

Remote Procedure Call (RPC) is mostly used for system-to-system communication, e.g microservice to microservice. If you just want a normal public API then you still want to go with REST. However, if you have an API which needs to poll for data from elsewhere, you can use RPC to communicate with that external service if that implements it too. Lots of large organisations use it for all their system communications, for example Google does this with an infrastructure called Stubby which includes an open source version you're probably already familiar with - gRPC.

Benefits of RPC:

Bi-directional streaming of data over HTTP/2 transport asynchronously.
Ability to generate both client & server stubs for most common languages.
Easier to serialise structured data.
Can be 3 - 10 times smaller than XML/JSON.
Can be anywhere from 20 - 100 times faster.

Set up & Prerequisites

Before we can do anything, we need to install a few things:

// Install Twirp protoc generator
go get github.com/twitchtv/twirp/protoc-gen-twirp

// Install Protoc - The protobuf compiler
https://github.com/google/protobuf/releases/tag/v3.6.0

// Mac users can install the above with HomeBrew
brew update && brew install protobuf

// Install Golang Protobuf support
go get -u github.com/golang/protobuf/protoc-gen-go

So it begins

Now that we have everything installed, we can crack on with the fun part, writing the code! First, we need to create our project directory and flesh out the structure:

cd $GOPATH/src/github.com/<your_username>
mkdir rpc-todo && cd $_
mkdir rpc
cd rpc && mkdir todo && cd $_

We're doing a couple of things here to separate our application structure and I'll explain each part as we progress through the tutorial. For now, let me explain the rpc directory. This is an important convention that sometimes appears under different namespaces; for example, you may see it named proto. The convention serves as a location for protobuf service definitions and their generated code. In our rpc directory we've created our first service directory todo . Now we can write out our service definition in a new file, service.proto:

syntax = "proto3";

package <your_github_username>.rpctodo.todo;
option go_package = "todo";

service Todo {
    rpc MakeTodo(Todo) returns (Empty);
    rpc GetTodo(TodoQuery) returns (Todo);
    rpc AllTodos(Empty) returns (TodoResponse);
    rpc RemoveTodo(TodoQuery) returns (Empty);
    rpc MarkComplete(TodoQuery) returns (Empty);
}

message TodoQuery {
    int32 id = 1;
}

message TodoResponse {
    repeated Todo todos = 1;
}

message Todo {
    int32 ID = 1;
    string title = 2;
    bool complete = 3;
}

message Empty {
}

There's a few things going on here so let's start at the top. First, we're declaring the syntax version of our protobuf file as version 3. Secondly, we declare the proto package. This works in the following format:

package <user/organisation>.<repository_name>.<service_name>;

// For example, mine is:
package williamhgough.commando.todo;

Then, we provide the name we wish to appear when importing the service as a Go package. Once all that is done, we can begin defining our service. Think of this as defining the interface for our API. Each method in our service takes parameters and returns a value, these values must be of a message type. In the same way we're equating the service keyword to an interface, we can think of a message as a struct type in Go. We are simply defining a new type and outlining its fields. There are some clear differences though. You may have noticed the = 1, = 2, = 3 after each message field, these represent the order of the fields. This allows us to come back later and add more fields or change the order.

It's also important to take note of both the Empty message type and the keyword repeated inside our TodoResponse. Since protobuf does not use the assumed return types void or null, we must define Empty instead. Inside TodoResponse, the repeated key word is used to return more than one item of type Todo. Since protobufs don't have array types for comparison in a Golang struct, you would have used a slice of Todo like so []Todo. For more information on protobuf syntax, check out the documentation here.

Now that we understand our way around a proto definition, let's generate our Golang service code using the protoc command. In the root of the project, execute the following:

protoc --proto_path=$GOPATH/src:. --twirp_out=. --go_out=. ./rpc/todo/service.proto

You should now see, inside your rpc/todo service directory, two new files service.pb.go and service.twirp.go. These are generated files. Do not edit them! Twirp and the Protoc compiler have generated client and server stubs for interacting with your RPC service.

We're about to make use of the generated files so take a minute to read through them and inspected the generated Go code. The next step for us is to implement our Todo service interface. In the root of your project directory, execute the following:

mkdir internal && cd $_
mkdir todoserver && cd $_
touch server.go

Here we create our internal directory, this is to hold our service implementations, one directory for each, as you can see we've named ours todoserver. Inside the server.go file you just created, let's add the following code:

package todoserver

import (
    "context"
    "github.com/twitchtv/twirp"
    // Import our protobuf package, namespace it to pb. This is a convention!
    pb "github.com/<your_username>/rpc-todo/rpc/todo"
)

// Create our Server type
type Server struct{}

// Seed our application with one To-Do to start with.
// The map holds the To-Do ID as the key to a To-Do value.
var todos = map[int]*pb.Todo{
    1: &pb.Todo{ID: 1, Title: "hello world", Complete: false},
}

// MakeTodo creates a To-Do
func (s *Server) MakeTodo(ctx context.Context, todo *pb.Todo) (*pb.Empty, error) { return nil, nil }

// GetTodo returns a single To-Do for given ID.
func (s *Server) GetTodo(ctx context.Context, query *pb.TodoQuery) (*pb.Todo, error) { return nil, nil }

// AllTodos returns all To-Dos.
func (s *Server) AllTodos(ctx context.Context, e *pb.Empty) (*pb.TodoResponse, error) { return nil, nil }

// RemoveTodo deletes the To-Do with the given ID.
func (s *Server) RemoveTodo(ctx context.Context, q *pb.TodoQuery) (*pb.Empty, error) { return nil, nil }

// MarkComplete sets the To-Do with given ID to completed.
func (s *Server) MarkComplete(ctx context.Conext, q *pb.TodoQuery) (*pb.Empty, error) { return nil, nil }

The code above creates a new Server type and makes it implement our Todo service interface. For now, our code just satisfies the interface and won't actually function. Now we have the base implementation, let's finish them off with the actual code we need. Make sure you read through the comments in the code as they should explain what is happening at each point:

func (s *Server) MakeTodo(ctx context.Context, todo *pb.Todo) (*pb.Empty, error) {
    // Validate the given ID, return error if not > 0
    if todo.ID < 1 {
        return nil, twirp.InvalidArgumentError("ID:", "No ID provided")
    }
    // Validate Title, should not add empty To-Do.
    if todo.Title == "" {
        return nil, twirp.InvalidArgumentError("Title:", "No title given")
    }
    // Set To-Do in the Todos map, passing ID as key.
    todos[todo.ID] = todo
    // Return Empty response and nil error.
    return &pb.Empty{}, nil
}

func (s *Server) GetTodo(ctx context.Context, q *pb.TodoQuery) (*pb.Todo, error) {
    // Validate the given ID, return error if not > 0
    if q.Id < 1 {
        return nil, twirp.InvalidArgumentError("ID:", "Must be greater than zero")
    }
    // Return To-Do with given ID from map and nil error
    return todos[q.Id], nil
}


func (s *Server) AllTodos(ctx context.Context, e *pb.Empty) (*pb.TodoResponse, error) {
    // Create new slice of To-Dos
    t := []*pb.Todo{}
    // for each one in our map, add to slice
    for _, v := range todos {
        t = append(t, v)
    }
    // If there are no todos, return error
    if len(t) < 1 {
        return nil, twirp.InternalError("No todos found!")
    }
    // Otherwise return Todos and nil error
    return &pb.TodoResponse{
        Todos: t,
    }, nil
}

func (s *Server) RemoveTodo(ctx context.Context, q *pb.TodoQuery) (*pb.Empty, error) {
    // Validate ID is greater than 0
    if q.Id < 1 {
        return nil, twirp.InvalidArgumentError("ID:", "Must be greater than zero")
    }
    // Remove To-Do with given ID from map
    delete(todos, q.Id)
    // Both return nil
    return &pb.Empty{}, nil
}

func (s *Server) MarkComplete(ctx context.Context, q *pb.TodoQuery) (*pb.Empty, error) {
    // Validate ID greater than 1
    if q.Id < 1 {
        return nil, twirp.InvalidArgumentError("ID:", "Must be greater than zero")
    }
    // Find To-Do with given ID in Todos map and set complete to true
    todos[q.Id].Complete = true
    // Return nil for both
    return &pb.Empty{}, nil
}

Alright, we're almost there. The final stage of this application involves building our CLI programs. We need a program to run the server and a CLI to interact with it. Both applications are very straightforward. Let's go ahead and create the server. In the project root, execute the following:

mkdir cmd && cd $_
mkdir todo-server && cd $_
touch main.go

Here we've created our client application directory and the main program. If you aren't familiar with Go application structure and using the cmd directory, it's a simple pattern to structure applications where a command-line interface is used. We name the directories inside cmd with the desired program name so that when we run go build, we get a binary named todo-server instead of main. Inside main.go, add the following code:

package main

import (
    "fmt"
    "net/http"

    "github.com/<your_username>/rpc-todo/internal/todoserver"
    pb "github.com/<your_username>/rpc-todo/rpc/todo"
)

func main() {
    // Create new Server from our server package.
    server := &todoserver.Server{}
    // Create Twirp Handler for our server from generated server stub.
    // This takes the form pb.New<SERVICE_NAME>Server()
    twirpHandler := pb.NewTodoServer(server, nil)

    // Start HTTP server on 8080, passing our twirpHandler as the servemux.
    http.ListenAndServe(":8080", twirpHandler)
    fmt.Println("To-Do server listening on :8080")
}

Next, we create the client program:

cd cmd
mkdir todo && cd $_
touch main.go

The client application has more functionality than our server as we need to parse any command line flags, validate them, and then execute the desired process. All of which can be achieved with the flags package and a simple switch statement. Nice and easy:

package main

import (
    "context"
    "flag"
    "fmt"
    "net/http"

    "github.com/<your_username>/rpc-todo/rpc/todo"
)

// Create global To-Do counter
var todoCount int32 = 1

func main() {
    // Define our interaction flags.
    action := flag.String("command", "list", "Command to interact with Todos. create, get, list, delete")
    title := flag.String("title", "", "The todo title")
    id := flag.Int("id", 1, "The ID of the todo you wish to retrieve")
    flag.Parse()

    // Create the generated Twirp Protobuf Client for our Todo service
    client := todo.NewTodoProtobufClient("http://localhost:8080", &http.Client{})

    switch *action {
    case "create":
        _, err := client.MakeTodo(context.Background(), &todo.Todo{
            ID:       todoCount + 1,
            Title:    *title,
            Complete: false,
        })
        if err != nil {
            fmt.Println("could not create todo:", err)
        }
    case "list":
        res, err := client.AllTodos(context.Background(), &todo.Empty{})
        if err != nil {
            fmt.Println("could not fetch todos:", err)
        }
        for i, t := range res.Todos {
            fmt.Printf("%d. %s [%v]\n", i+1, t.Title, t.Complete)
        }
    case "get":
        todo, err := client.GetTodo(context.Background(), &todo.TodoQuery{
            Id: int32(*id),
        })
        if err != nil {
            fmt.Println("could not fetch todo:", err)
        }

        fmt.Printf("%d. %s [%v]\n", todo.ID, todo.Title, todo.Complete)
    case "delete":
        _, err := client.RemoveTodo(context.Background(), &todo.TodoQuery{
            Id: int32(*id),
        })
        if err != nil {
            fmt.Println("could not remove todo:", err)
        }
    case "complete":
        _, err := client.MarkComplete(context.Background(), &todo.TodoQuery{
            Id: int32(*id),
        })
        if err != nil {
            fmt.Println("could not complete todo:", err)
        }
    default:
        fmt.Println("invalid command, please try again.")
    }
}

Voila! That's it! See how we automagically generated Twirp client uses the same API as our server code? This makes it super intuitive to use and interact with. All that remains for us to do is to verify it's all working in our command line. Open up two terminal windows in your project root:

Window 1 - Start the To-Do service server.

cd cmd/todo-server
go run main.go

// Output
fmt.Println("To-Do server listening on :8080")

Window 2 - Interact with To-Do client.

cd cmd/todo

// list To-Dos
go run main.go
1. hello world [false]

// create To-Do
go run main.go -command="create" -title="Hello DevTheWeb"

// get specific To-Do
go run main.go -command="get" -id=2
2. Hello DevTheWeb [false]

// Complete To-Do
go run main.go -command="complete" -id=1

// Remove To-Do
go run main.go -command="delete" -id=1

And that's a wrap! You've done it! You can now add basic RPC to your repertoire of skills.

Okay, so that was a lot to take in and hopefully, I've been able to explain in a way that leaves you feeling clear and confident! I'd also like to recommend trying out Google's gRPC library. I opted for Twirp here because I find it more intuitive and easy to use. The best advice I could give you is to go ahead and try writing another RPC service with this freshly acquired knowledge. Maybe a library for finding books or a movies search tool using OMDB? There is an undeniable truth in the old proverb: