Load balancing gRPC service with Nginx

So far we’ve learned a lot about how to develop backend web services with gRPC. When it comes to deployment, one important thing we should consider is load balancing.

A large scale gRPC deployment typically has a number of identical backend servers and a number of clients. Load balancing is used for distributing the load from clients optimally across available servers.

Here's the link to the full gRPC course playlist on Youtube
Gitlab repository: pcbook-go and pcbook-java

Table of contents

• Types of load balancing
  • Server-side load balancing
  • Client-side load balancing
  • Pros and cons
• Code refactoring
  • Update server
  • Update client
  • Test the new flag
• Update Makefile
• Install Nginx
• Config Nginx for insecure gRPC
• Config Nginx for gRPC with TLS
  • Enable TLS on Nginx but keep gRPC servers insecure
  • Enable TLS on both Nginx and gRPC servers
  • Multiple routing locations

Types of load balancing

There are 2 main options for gRPC load balancing: server-side and client-side. Deciding which one to use is a primary architectural choice.

Server side load balancing

In server-side load balancing, the client issues RPCs to a load balancer or proxy, such as Nginx or Envoy. The load balancer distributes the RPC call to one of the available backend servers.

It also keeps track of load on each server and implements algorithms for distributing load fairly. The clients themselves do not know about the backend servers.

Client side load balancing

In client-side load balancing, the client is aware of multiple backend servers and chooses one to use for each RPC. Usually, the backend servers register themselves with a service discovery infrastructure, such as Consul or Etcd. Then the client communicates with that infrastructure to know the addresses of the servers.

A thick client implements the load balancing algorithms itself. For example, in a simple configuration, where the server load is not considered, the client can just round-robin between available servers.

Another approach is to use a look-aside load balancer, where the load balancing smarts are implemented in a special load-balancing server. Clients query the look-aside load balancer to get the best server(s) to use. The heavy lifting of keeping server state, service discovery, and implementation of a load balancing algorithm is consolidated in the look-aside load balancer.

Pros and cons

One pros of server-side load balancing is simple client implementation. All client needs to know is the address of the proxy, no more coding is needed. This approach works even for untrusted clients, which means the gRPC service can be open for everyone from the public internet.

However, its cons is adding 1 more extra hop to the call. All RPCs have to go through the proxy before reaching the backend server, thus causing higher latency. Therefore, this server-side load balancing is suitable for the cases where there are many clients, possibly untrusted from the open internet who want to connect to our gRPC servers in a data center.

The client-side load balancing, on the other hand, doesn’t add any extra hop to the call and thus giving us higher performance in general. However, the client implementation now becomes complex, especially for thick client approach. Therefore, it should only be used for trusted clients, or we will need to use a look-aside load balancer to stand in the front of the trust boundary network. Client-side load balancing is often used in a very high traffic system and microservices architecture.

In this article, we will learn how to set up server-side load balancing for our gRPC services with Nginx.

Code refactoring

Since I’m gonna show you different Nginx configurations where TLS can be enabled or disabled on the server and client, let’s update our code a bit to take in a new command-line argument.

Update server

On the server, let's add a new boolean flag enableTLS, which will tell us whether we want to enable TLS on our gRPC server or not. Its default value is false.

func main() {
    port := flag.Int("port", 0, "the server port")
    enableTLS := flag.Bool("tls", false, "enable SSL/TLS")

    flag.Parse()
    log.Printf("start server on port %d, TLS = %t", *port, *enableTLS)

    ...
}

Then let's extract the interceptors to a separate serverOptions variable. We check the enableTLS flag. Only in case it’s enabled then we load the TLS credentials, and append that credentials to the server options slice. Finally we just pass the server options to the grpc.NewServer() function call.

func main() {
    ...

    interceptor := service.NewAuthInterceptor(jwtManager, accessibleRoles())
    serverOptions := []grpc.ServerOption{
        grpc.UnaryInterceptor(interceptor.Unary()),
        grpc.StreamInterceptor(interceptor.Stream()),
    }

    if *enableTLS {
        tlsCredentials, err := loadTLSCredentials()
        if err != nil {
            log.Fatal("cannot load TLS credentials: ", err)
        }

        serverOptions = append(serverOptions, grpc.Creds(tlsCredentials))
    }

    grpcServer := grpc.NewServer(serverOptions...)

    ...
}

And that’s it for the server. Let’s do similar thing for the client!

Update client

First we add the enableTLS flag to the command line argument. Then we define a transportOption variable with the default value grpc.WithInsecure().

func main() {
    serverAddress := flag.String("address", "", "the server address")
    enableTLS := flag.Bool("tls", false, "enable SSL/TLS")

    flag.Parse()
    log.Printf("dial server %s, TLS = %t", *serverAddress, *enableTLS)

    transportOption := grpc.WithInsecure()

    if *enableTLS {
        tlsCredentials, err := loadTLSCredentials()
        if err != nil {
            log.Fatal("cannot load TLS credentials: ", err)
        }

        transportOption = grpc.WithTransportCredentials(tlsCredentials)
    }

    cc1, err := grpc.Dial(*serverAddress, transportOption)
    if err != nil {
        log.Fatal("cannot dial server: ", err)
    }

    authClient := client.NewAuthClient(cc1, username, password)
    interceptor, err := client.NewAuthInterceptor(authClient, authMethods(), refreshDuration)
    if err != nil {
        log.Fatal("cannot create auth interceptor: ", err)
    }

    cc2, err := grpc.Dial(
        *serverAddress,
        transportOption,
        grpc.WithUnaryInterceptor(interceptor.Unary()),
        grpc.WithStreamInterceptor(interceptor.Stream()),
    )
    if err != nil {
        log.Fatal("cannot dial server: ", err)
    }

    laptopClient := client.NewLaptopClient(cc2)
    testRateLaptop(laptopClient)
}

Only when the enableTLS flag value is true, we load the TLS credentials from PEM files and change the transportOption to grpc.WithTransportCredentials(tlsCredentials). Finally we pass the transportOption to the grpc connections. And the client is done.

Test the new flag

Now if we run make server, we can see that the server is running with TLS disabled.

And if we run make client, it’s also running with no TLS, and all the RPC calls are successful.

If we add -tls flag to the make server command, and restart it, the TLS will be enabled.

...

server:
    go run cmd/server/main.go -port 8080 -tls

...

If we run make client now, the requests will fail:

We have to enable TLS on client side as well, by adding -tls flag to the make client command.

...

client:
    go run cmd/client/main.go -address 0.0.0.0:8080 -tls

...

And now we can see the requests are successful again.

Update Makefile

Alright, now the TLS flag is working as we wanted. Before adding Nginx, let's update our Makefile a bit so that we can easily run multiple instances of the server and the client with or without TLS.

I’m gonna remove the -tls flags so that the make server and make client commands will run without TLS. And I will add 2 more make commands to run 2 instances of the server on different ports. Let’s say the 1st server will run on port 50051, and the 2nd sever will run on port 50052.

...

server:
    go run cmd/server/main.go -port 8080

client:
    go run cmd/client/main.go -address 0.0.0.0:8080

server1:
    go run cmd/server/main.go -port 50051

server2:
    go run cmd/server/main.go -port 50052

...

Let’s also add 3 more make commands to start the client and servers with TLS. The client-tls command will run the client with TLS. The make server1-tls command will start a TLS server on port 50051, and the make server2-tls command will start another TLS server on port 50052.

...

client-tls:
    go run cmd/client/main.go -address 0.0.0.0:8080 -tls

server1-tls:
    go run cmd/server/main.go -port 50051 -tls

server2-tls:
    go run cmd/server/main.go -port 50052 -tls

...

Install Nginx

The next thing we need to do is to install Nginx. Since I’m on a mac, I can simply use Homebrew:

❯ brew install nginx

After nginx is installed, we can go to this usr/local/etc/nginx folder to config it. Let's open the nginx.conf file with with visual studio code.

❯ cd /usr/local/etc/nginx
❯ code nginx.conf

This is the default configuration:

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    server {
        listen       8080;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            root   html;
            index  index.html index.htm;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }

    include servers/*;
}

There are several things that we don’t need to care about in this tutorial, so let's update this config file.

Config Nginx for insecure gRPC

First, let's remove the user config, uncomment the error log, remove the config for log levels and process id, and let’s say we just need 10 worker connections for now.

One important thing we need to do is to config the right location to store the error log and access log files. In my case, Homebrew has already created a log folder for Nginx at /usr/local/var/log/nginx, so I just go ahead and use it in the error/access log setting.

worker_processes  1;

error_log  /usr/local/var/log/nginx/error.log;

events {
    worker_connections  10;
}

http {
    access_log  /usr/local/var/log/nginx/access.log;

    server {
        listen       8080 http2;

        location / {
        }
    }
}

Now in the server block, we have a listen command to listen to incoming requests from client on port 8080. This is the default config for a normal HTTP server. Since gRPC uses HTTP/2, we should add http2 at the end of this command.

Let’s remove the server name and charset since we don’t need them now. Similar for the access log because we’ve already defined it above. Let’s also delete the config for default root HTML file and everything after the location block as we don’t care about them for now.

OK, now we want to load balance the incoming requests to our 2 server instances. So we should define a upstream for them. I’m gonna call it upstream pcbook_services. Inside this block, we use the server keyword to declare a server instance. The first one is running on localhost port 50051, and the second one is running on port 50052.

worker_processes  1;

error_log  /usr/local/var/log/nginx/error.log;

events {
    worker_connections  10;
}

http {
    access_log  /usr/local/var/log/nginx/access.log;

    upstream pcbook_services {
        server 0.0.0.0:50051;
        server 0.0.0.0:50052;
    }

    server {
        listen       8080 http2;

        location / {
            grpc_pass grpc://pcbook_services;
        }
    }
}

Then to route all RPC calls to the upstream, in the location block, we use grpc_pass keyword, followed by the grpc:// scheme and the name of the upstream, which is pcbook_services.

And that’s it! The load balancing for our insecure gRPC server is done.

Let’s run nginx in the terminal to start it.

❯ nginx
❯ ps aux | grep nginx
quangpham         9013   0.0  0.0  4408572    800 s000  S+    6:13PM   0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox nginx
quangpham         9007   0.0  0.0  4562704   1124   ??  S     6:12PM   0:00.00 nginx: worker process
quangpham         9006   0.0  0.0  4422416    612   ??  Ss    6:12PM   0:00.00 nginx: master process nginx

We can check if it’s running or not using the ps and grep command. Let’s check out the log folder:

❯ cd /usr/local/var/log/nginx
❯ ls -l
total 0
-rw-r--r--  1 quangpham  admin  0 Oct 11 18:12 access.log
-rw-r--r--  1 quangpham  admin  0 Oct 11 18:12 error.log

As you can see, 2 log files are generated: access.log and error.log. They’re empty at the moment because we haven’t sent any requests yet.

Now let’s run make server1 to start the first server on port 50051 with TLS = false. Then on another tab, run make server2 to start the second server on port 50052, also with TLS disabled. Finally, we run make client on another new tab.

Looks good. All RPC calls are successful. Let’s check the logs on our servers.

The server2 receives 2 create laptop requests.

And the server1 receives 1 login request and 1 create laptop request. Excellent!

And after a while, there’s another login request comming to this server. That’s because our client is still running, and it periodically calls login to refresh the token.

I hope you still remember the codes that we’ve written in the gRPC interceptors lecture.

OK, now let’s look at the nginx access log file.

You can see first there’s a login call, then 3 create laptop calls, and finally another login call. So everything is working exactly as we expect.

Next I’m gonna show you how to enable SSL/TLS for Nginx.

Config Nginx for gRPC with TLS

In a typical deployment, the gRPC servers are already running inside a trusted network, and only the load balancer (Nginx in this case) is exposed to the public internet. So we can leave our gRPC servers running without TLS as before, and only add TLS to Nginx.

Enable TLS on Nginx but keep gRPC servers insecure

To do that, we will need to copy 3 pem files to the nginx config folder:

• The server’s certificate
• The server’s private key
• And the certificate of the CA who signed client’s certificate if we use mutual TLS.

OK, so now I’m gonna cd to the /usr/local/etc/nginx folder and create a new cert folder. Then I will copy those 3 pem files from our pcbook project to this folder.

❯ cd /usr/local/etc/nginx
❯ mkdir cert
❯ cp ~/Projects/techschool/pcbook-go/cert/server-cert.pem cert
❯ cp ~/Projects/techschool/pcbook-go/cert/server-key.pem cert
❯ cp ~/Projects/techschool/pcbook-go/cert/ca-cert.pem cert

Alright, now all the certificate and key files are ready. Let's go back to our nginx config file.

To enable TLS, we first need to add ssl to the listen command. Then we use the ssl_certificate command to give Nginx the location of the server’s certificate file. And use the ssl_certificate_key command to give it the location of the server’s private key file.

...

    server {
        listen       8080 ssl http2;

        ssl_certificate cert/server-cert.pem;
        ssl_certificate_key cert/server-key.pem;

        ssl_client_certificate cert/ca-cert.pem;
        ssl_verify_client on;

        location / {
            grpc_pass grpc://pcbook_services;
        }
    }

...

As we’re using mutual TLS, we also need to use the ssl_client_certificate command to tell nginx the location of the client CA’s certificate file. And finally we set ssl_verify_client to on to tell nginx to verify the authenticity of the certificate that client will send.

And we’re done. Let’s restart nginx. We run nginx -s stop to stop it first. Then we start it with nginx command.

❯ nginx -s stop
❯ nginx

Our server is already running, so let’s run the client!

If we just run make client, it will run without TLS, so the request will fail, because Nginx is now running with TLS enabled.

Now let’s run make client-tls.

This time the client is running with TLS, and all requests are successful.

Keep in mind that our servers are still running without TLS. So basically what happens is: only the connection between client and Nginx is secure, and Nginx is connecting to our backend servers via another insecure connection.

Once Nginx receives the encrypted data from client, it will decrypt the data before forwarding it to the backend servers. Therefore, you should only use this approach if the Nginx and backend servers stay in the same trusted network.

OK, but what if they are not on the same trusted network? Well, in that case, we have no choice but to enable TLS on our backend servers as well, and config nginx to work with it.

Enable TLS on both Nginx and gRPC servers

Let’s stop the current server1 and server2, then restart them with TLS.

❯ make server1-tls
❯ make server2-tls

Now if we run make client-tls immediately, the request will fail.

The reason is, although the TLS handshake between the client and Nginx succeeded, the TLS handshake between Nginx and our backend servers failed since the backend servers are now expecting a secure TLS connection, while Nginx is still using an insecure connection when connecting to the backend servers.

As you can see in the error log, the failure happened when Nginx talked to the upstream servers.

To enable secure TLS connection between nginx and upstream, in the nginx.conf file, we have to change the grpc scheme to grpcs.

...

    server {
        ...

        location / {
            grpc_pass grpcs://pcbook_services;
        }
    }

...

This should be enough if we just use server-side TLS. However, in this case, we’re using mutual TLS, so if we just restart Nginx now and rerun make client-tls, the request will still fail because Nginx is not configured to send its certificate to the upstream servers yet.

We’ve got the bad certificate error as you can see in the log.

Let’s see what happens if we go to the server code cmd/server/main.go and change the ClientAuth field from tls.RequireAndVerifyClientCert to tls.NoClientCert, which means we will just use server-side TLS.

func loadTLSCredentials() (credentials.TransportCredentials, error) {
    ...

    // Create the credentials and return it
    config := &tls.Config{
        Certificates: []tls.Certificate{serverCert},
        ClientAuth:   tls.NoClientCert,
        ClientCAs:    certPool,
    }

    return credentials.NewTLS(config), nil
}

Then restart server1-tls and server2-tls, and run make client-tls again.

❯ make server1-tls
❯ make server2-tls
❯ make client-tls

This time all requests are successful. That's exactly what we expected!

OK now what if we really want mutual TLS between nginx and upstream?

Let’s change the ClientAuth field back to tls.RequireAndVerifyClientCert, restart the 2 TLS backend servers, and get back to our nginx.conf file.

This time, we must instruct Nginx to do mutual TLS with the backend servers by giving it the location of the certificate and private key. We use the grpc_ssl_certificate keyword for the certificate, and the grpc_ssl_certificate_key keyword for the private key.

You can generate a different pair of certificate and private key for Nginx if you want. Here I simply use the same certificate and private key of the servers.

worker_processes  1;

error_log  /usr/local/var/log/nginx/error.log;

events {
    worker_connections  10;
}

http {
    access_log  /usr/local/var/log/nginx/access.log;

    upstream pcbook_services {
        server 0.0.0.0:50051;
        server 0.0.0.0:50052;
    }

    server {
        listen       8080 ssl http2;

        # Mutual TLS between gRPC client and nginx
        ssl_certificate cert/server-cert.pem;
        ssl_certificate_key cert/server-key.pem;

        ssl_client_certificate cert/ca-cert.pem;
        ssl_verify_client on;

        location / {
            grpc_pass grpcs://pcbook_services;

            # Mutual TLS between nginx and gRPC server
            grpc_ssl_certificate cert/server-cert.pem;
            grpc_ssl_certificate_key cert/server-key.pem;
        }
    }
}

OK, let’s try it.

First stop the current nginx process, then start a new one. And run make client-tls again.

❯ nginx -s stop
❯ nginx
❯ make client-tls

This time all requests are successful. Perfect!

Multiple routing locations

There’s one more thing I want to show you before we finish.

As you’ve already seen, the login and create-laptop requests are now evenly distributed between our 2 backend servers. But sometimes, we might want to separate the authentication service and the business logic service.

For example, let’s say we want all login requests to go to server 1, and all other requests to go to server 2. In that case, we can also tell Nginx to route the requests based on its path.

worker_processes  1;

error_log  /usr/local/var/log/nginx/error.log;

events {
    worker_connections  10;
}

http {
    access_log  /usr/local/var/log/nginx/access.log;

    upstream auth_services {
        server 0.0.0.0:50051;
        server 0.0.0.0:50052;
    }

    upstream laptop_services {
        server 0.0.0.0:50051;
        server 0.0.0.0:50052;
    }

    server {
        listen       8080 ssl http2;

        # Mutual TLS between gRPC client and nginx
        ssl_certificate cert/server-cert.pem;
        ssl_certificate_key cert/server-key.pem;

        ssl_client_certificate cert/ca-cert.pem;
        ssl_verify_client on;

        location /techschool.pcbook.AuthService {
            grpc_pass grpcs://auth_services;

            # Mutual TLS between nginx and gRPC server
            grpc_ssl_certificate cert/server-cert.pem;
            grpc_ssl_certificate_key cert/server-key.pem;
        }

        location /techschool.pcbook.LaptopService {
            grpc_pass grpcs://laptop_services;

            # Mutual TLS between nginx and gRPC server
            grpc_ssl_certificate cert/server-cert.pem;
            grpc_ssl_certificate_key cert/server-key.pem;
        }
    }
}

Here I just copy the path /techschool.pcbook.AuthService of the AuthService and paste it to this location. Then I change this upstream name to auth_services. It should only connect to the server1 at port 50051.

Then I add another upstream for the laptop_services and make it connect to only server2 at port 50052. Then duplicate the location block, change the upstream name to laptop_services, and update the path to techschool.pcbook.LaptopService.

OK, let’s try this! We just need to restart Nginx and run make client-tls.

Now we can see only login request goes to server1.

And all other create laptop requests go to server2. Even if we run this make client-tls multiple times.

So it works! And that wraps up our lecture about load balancing gRPC with Nginx.

I’m gonna push this nginx config file to the pcbook-go repository so that you can download and play with it if you like.

Thanks a lot for reading and following the course. Happy coding and see you in the next one!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.