Deploy Nginx Load Balancer for Rancher

Introduction

In the dynamic landscape of container orchestration, efficient load balancing is paramount. As organizations adopt Kubernetes and Rancher rke2 for managing their workloads, the need for robust load balancing solutions becomes increasingly critical. Nginx is a versatile web server and reverse proxy that can also serve as a powerful Layer 4 load balancer.

In this article, we delve into the intricacies of leveraging Nginx to distribute incoming requests across control plane nodes in your Kubernetes cluster. By harnessing Nginx’s stream module, we unlock the ability to perform Layer 4 load balancing, ensuring optimal traffic distribution and high availability for our applications and also a fixed registration address for rke2 nodes.

Prerequisites

we've installed Nginx on a Debian 12 with IP address 192.168.100.100. We also have 3 Server(Control plane nodes) and 3 Agent(Worker) nodes. In total, our machines are listed here:

• Control-Plane-1 => IP: 192.168.100.11 , OS: Debian 12 , Hostname: kuber-master-1

• Control-Plane-2 => IP: 192.168.100.12 , OS: Debian 12 , Hostname: kuber-master-2

• Control-Plane-3 => IP: 192.168.100.13 , OS: Debian 12 , Hostname: kuber-master-3

• Worker-1 => IP: 192.168.100.14 , OS: Debian 12 , Hostname: kuber-worker-1

• Worker-2 => IP: 192.168.100.15 , OS: Debian 12 , Hostname: kuber-worker-2

• Worker-3 => IP: 192.168.100.16 , OS: Debian 12 , Hostname: kuber-worker-3

• L4 Load Balancer => ip: 192.168.100.100
  
  

Install and Enable Stream module

To use Nginx as a layer 4 load balancer, it must have the stream module included or be able to dynamically use it. To check if Nginx can use Stream module, check Nginx configured modules:

nginx -V

The output will be something like below:

configure arguments: --with-cc-opt='-g -O2
-ffile-prefix-map=/build/nginx-AoTv4W/nginx-1.22.1=.
-fstack-protector-strong -Wformat -Werror=format-security
-fPIC -Wdate-time -D_FORTIFY_SOURCE=2'
--with-ld-opt='-Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf
--http-log-path=/var/log/nginx/access.log
--error-log-path=stderr --lock-path=/var/lock/nginx.lock
--pid-path=/run/nginx.pid
--modules-path=/usr/lib/nginx/modules
--http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi
--http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat
--with-debug --with-pcre-jit --with-http_ssl_module
--with-http_stub_status_module --with-http_realip_module
--with-http_auth_request_module --with-http_v2_module
--with-http_dav_module --with-http_slice_module
--with-threads --with-http_addition_module
--with-http_flv_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_mp4_module
--with-http_random_index_module
--with-http_secure_link_module --with-http_sub_module
--with-mail_ssl_module --with-stream_ssl_module
--with-stream_ssl_preread_module --with-stream_realip_module --with-http_geoip_module=dynamic
--with-http_image_filter_module=dynamic
--with-http_perl_module=dynamic
--with-http_xslt_module=dynamic --with-mail=dynamic
--with-stream=dynamic --with-stream_geoip_module=dynamic

Check for the option --with-stream=dynamic. This argument is necessary as it allows us to use the stream module in Nginx.

NOTE: If you can't find the --with-stream=dynamic option, you've to recompile and install Nginx with this option.

Now that Nginx supports stream module, we must install this module so as to use it in Nginx. Install the stream module using the command below:

apt-get install -y libnginx-mod-stream

After the module is installed, we should include it in the Nginx config file. Go to /etc/nginx/nginx.conf and include it:

user nginx;
worker_processes  auto;

load_module modules/ngx_stream_module.so;
### Other lines are emitted

The modules are usually located in /usr/lib/nginx/modules. If you want to verify it, use nginx -V and check the output. There is the option --modules-path that indicates the modules path.

Configure Stream in Nginx

Now that the stream module has been installed and imported, it's time to use it.
At the end of the Nginx main config file in /etc/nginx/nginx.conf add the following lines:

stream {
        log_format basic '$remote_addr [$time_local] '
                     '$protocol $status $bytes_sent $bytes_received '
                     '$session_time';
        include /etc/nginx/conf.d/*.conf;
}

Here we've specified the log format of the stream module. It's also been specified that files in /etc/nginx/conf.d ending with .conf are operated in layer 4 and not layer 7.

As the Nginx instance is supposed to be used for Rancher and Kubernetes, it should have two upstreams, one for rke2 and one for the kube-api server.

First of all, we add configuration for Rancher. Create /etc/nginc/conf.d/rke.conf and add the following lines:

upstream rancher_nodes {
        server 192.168.100.11:9345 fail_timeout=1s max_fails=2;
        server 192.168.100.12:9345 fail_timeout=1s max_fails=2;
        server 192.168.100.13:9345 fail_timeout=1s max_fails=2;
}

server {
    listen 9345;

    access_log  /var/log/nginx/domains/rke/access.log basic;

    proxy_pass rancher_nodes;

}

The rke2 service listens on port 9345 and here Nginx proxy passes to port 9345 on Control plane nodes.

The next part will be the Kubernetes part. Again create /etc/nginc/conf.d/k8s.conf and add the below content:

upstream kubernetes_nodes {
        server 192.168.100.11:6443 fail_timeout=1s max_fails=2;
        server 192.168.100.12:6443 fail_timeout=1s max_fails=2;
        server 192.168.100.13:6443 fail_timeout=1s max_fails=2;
}

server {
    listen 6443;

    access_log  /var/log/nginx/domains/k8s/access.log basic;

    proxy_pass kubernetes_nodes;

}

The kube-api server operates on port 6443 So we've used reverse proxy for these ports on Control plane nodes.

Run Nginx

Test Nginx configuration:

nginx -t

If everything is ok, Restart the Nginx service:

systemctl restart nginx

Now if you check open ports, there must be 6443 and 9345 present.

ss -ntlp

Congratulations! Nginx from now on, should listen and load balance your requests to Control planes.

Summary

In this article we made Nginx to operate as a layer 4 load balancer with the help of Stream module.
We also covered how to use Nginx as a fixed registration address for Rancher rke2 and Kubernetes plus load balancing the requests to the Kubernetes API server.
To get more info about using Nginx and integrating it with Rancher rke2, please refer to the official documentation of Nginx and Rancher:

• https://nginx.org/en/docs/stream/ngx_stream_core_module.html
• https://docs.rke2.io/

You could also visit my other article about deploying Rancher rke2 with Cilium and Metallb if you're interested:
https://dev.to/arman-shafiei/install-rke2-with-cilium-and-metallb-48a4

Thank you for reading this article and please leave a comment if you have any suggestions or figured out an issue with this post.