DEV Community

Franck Pachot for AWS Heroes

Posted on

YugabyteDB Helm Chart services: LoadBalancer and headless ClusterIP 鈽革笍馃殌

Installing YugabyteDB with the Helm Chart creates 4 services. Here is an example where I deployed in AWS as in this previous blog post:

$ kubectl get services -n yb-demo-eu-west-1a

NAME                 TYPE           CLUSTER-IP       PORT(S)
yb-master-ui         LoadBalancer   10.100.227.70    7000:31261/TCP
yb-masters           ClusterIP      None             7000/TCP,7100/TCP
yb-tserver-service   LoadBalancer   10.100.31.106   6379:31669/TCP,9042:31874/TCP,5433:31376/TCP
yb-tservers          ClusterIP      None             9000/TCP,12000/TCP,11000/TCP,13000/TCP,9100/TCP,6379/TCP,9042/TCP,5433/TCP
Enter fullscreen mode Exit fullscreen mode

There are two services (LoadBalancer and ClusterIP) for the two StatefulSets (yb-master, the contol plane, and yb-tservers, the data plane)

ClusterIP headless service

The ClusterIP ones are created with no IP which means that they are just DNS entries and not a proxy. This is a headless service that can be used to distribute the connections without an intermediate component, using the Kubernetes DNS.

From the cluster (cluster.local) in a namespace (yb-demo-eu-west-1c), I can connect to any pod in the yb-tserver StatefulSets with the hostname yb-tservers.yb-demo-eu-west-1a.svc.cluster.local

Round robin to one StatefulSets pods

Here is an example, running psql from any pod, and connecting multiple times to this address:

$ kubectl run -it --rm --restart=Never --image postgres psql -- \
  psql -h yb-tservers.yb-demo-eu-west-1a.svc.cluster.local      \
  -p 5433 -U yugabyte -c 'select inet_server_addr()'

 inet_server_addr
------------------
 192.168.11.13
(1 row)

$ kubectl run -it --rm --restart=Never --image postgres psql -- \
  psql -h yb-tservers.yb-demo-eu-west-1a.svc.cluster.local      \
  -p 5433 -U yugabyte -c 'select inet_server_addr()'

 inet_server_addr
------------------
 192.168.20.199
(1 row)
Enter fullscreen mode Exit fullscreen mode

The connections went to different nodes. Those are different PostgreSQL backends running in different pods, but they expose the same logical database because YugabyteDB is a Distributed SQL database.

The two IP addresses above are the two pods in the yb-tserver StatefulSets:

$ kubectl get pods -n yb-demo-eu-west-1a -o wide

NAME           READY   STATUS    RESTARTS   AGE    IP               NODE                                          NOMINATED NODE   READINESS GATES
yb-master-0    2/2     Running   0          174m   192.168.2.75     ip-192-168-4-117.eu-west-1.compute.internal   <none>           <none>
yb-tserver-0   2/2     Running   0          174m   192.168.20.199   ip-192-168-4-117.eu-west-1.compute.internal   <none>           <none>
yb-tserver-1   2/2     Running   0          174m   192.168.11.13    ip-192-168-4-117.eu-west-1.compute.internal   <none>           <none>
Enter fullscreen mode Exit fullscreen mode

Note that I deployed the StatefulSets in three Availability Zones. The ClusterIP service connects to one AZ and this is probably what you want to do from the application servers, so that they connect in the same AZ. This ensures the minimum latency and maximum availability.

Cluster aware Smart Drivers

However, if you use the YugabyteDB Smart Drivers they will discover all nodes in all AZ:

$ kubectl run -it --rm --restart=Never --image postgres psql -- \
>   psql -h yb-tservers.yb-demo-eu-west-1a.svc.cluster.local      \
>   -p 5433 -U yugabyte -c 'select host,zone from yb_servers()'
                             host                              |    zone
---------------------------------------------------------------+------------
 yb-tserver-0.yb-tservers.yb-demo-eu-west-1c.svc.cluster.local | eu-west-1c
 yb-tserver-1.yb-tservers.yb-demo-eu-west-1a.svc.cluster.local | eu-west-1a
 yb-tserver-0.yb-tservers.yb-demo-eu-west-1b.svc.cluster.local | eu-west-1b
 yb-tserver-1.yb-tservers.yb-demo-eu-west-1b.svc.cluster.local | eu-west-1b
 yb-tserver-1.yb-tservers.yb-demo-eu-west-1c.svc.cluster.local | eu-west-1c
 yb-tserver-0.yb-tservers.yb-demo-eu-west-1a.svc.cluster.local | eu-west-1a
(6 rows)
Enter fullscreen mode Exit fullscreen mode

This mean that if you want to constrain the connections to one AZ when using the Smart Drivers, you need to use the tolopogy key, like aws.eu-west-1.eu-west-1a for example.

Endpoints

The ClusterIP headless service yb-tservers connects to the yb-tserver pods, 192.168.11.13 and 192.168.20.199 in my case, for all ports exposed by the Table Servers, 5433 being the YSQL one, which is the PostgreSQL-compatible API.

$ kubectl describe service yb-tservers -n yb-demo-eu-west-1a

Name:              yb-tservers
Namespace:         yb-demo-eu-west-1a
Labels:            app=yb-tserver
                   app.kubernetes.io/managed-by=Helm
                   chart=yugabyte
                   component=yugabytedb
                   heritage=Helm
                   release=yb-demo
                   service-type=headless
Annotations:       meta.helm.sh/release-name: yb-demo
                   meta.helm.sh/release-namespace: yb-demo-eu-west-1a
Selector:          app=yb-tserver
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                None
IPs:               None
Port:              http-ui  9000/TCP
Endpoints:         192.168.11.13:9000,192.168.20.199:9000
Port:              http-ycql-met  12000/TCP
TargetPort:        12000/TCP
Endpoints:         192.168.11.13:12000,192.168.20.199:12000
Port:              http-yedis-met  11000/TCP
TargetPort:        11000/TCP
Endpoints:         192.168.11.13:11000,192.168.20.199:11000
Port:              http-ysql-met  13000/TCP
TargetPort:        13000/TCP
Endpoints:         192.168.11.13:13000,192.168.20.199:13000
Port:              tcp-rpc-port  9100/TCP
TargetPort:        9100/TCP
Endpoints:         192.168.11.13:9100,192.168.20.199:9100
Port:              tcp-yedis-port  6379/TCP
TargetPort:        6379/TCP
Endpoints:         192.168.11.13:6379,192.168.20.199:6379
Port:              tcp-yql-port  9042/TCP
TargetPort:        9042/TCP
Endpoints:         192.168.11.13:9042,192.168.20.199:9042
Port:              tcp-ysql-port  5433/TCP
TargetPort:        5433/TCP
Endpoints:         192.168.11.13:5433,192.168.20.199:5433
Session Affinity:  None
Events:            <none>
Enter fullscreen mode Exit fullscreen mode

LoadBalancer

In addition to the ClusterIP service, the Helm Chart creates a LoadBalancer. I can see it in the AWS console:

console

instances

Here is the description from Kubernetes:

$ kubectl describe service yb-tserver-service -n yb-demo-eu-west-1a

Name:                     yb-tserver-service
Namespace:                yb-demo-eu-west-1a
Labels:                   app=yb-tserver
                          app.kubernetes.io/managed-by=Helm
                          chart=yugabyte
                          component=yugabytedb
                          heritage=Helm
                          release=yb-demo
                          service-type=endpoint
Annotations:              meta.helm.sh/release-name: yb-demo
                          meta.helm.sh/release-namespace: yb-demo-eu-west-1a
Selector:                 app=yb-tserver
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.100.31.106
IPs:                      10.100.31.106
LoadBalancer Ingress:     a95638155644e470abd19e552bc8ab01-1510239497.eu-west-1.elb.amazonaws.com
Port:                     tcp-yedis-port  6379/TCP
TargetPort:               6379/TCP
NodePort:                 tcp-yedis-port  31498/TCP
Endpoints:                192.168.11.13:6379,192.168.20.199:6379
Port:                     tcp-yql-port  9042/TCP
TargetPort:               9042/TCP
NodePort:                 tcp-yql-port  30927/TCP
Endpoints:                192.168.11.13:9042,192.168.20.199:9042
Port:                     tcp-ysql-port  5433/TCP
TargetPort:               5433/TCP
NodePort:                 tcp-ysql-port  31565/TCP
Endpoints:                192.168.11.13:5433,192.168.20.199:5433
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
Enter fullscreen mode Exit fullscreen mode

The ingress host is the one I can use to connect from outside of the Kubernetes cluster. In this Multi-AZ configuration I have one per AZ. Let's check the round-robin with 100 sucessive connections:

$ for i in {1..100} ; do
   psql -h a95638155644e470abd19e552bc8ab01-1510239497.eu-west-1.elb.amazonaws.com -p 5433 \
   -t -A -c '
     select inet_server_addr()
    ' ; done | sort | uniq -c

     52 192.168.11.13
     48 192.168.20.199

Enter fullscreen mode Exit fullscreen mode

48% on yb-tserver-0 and 52% on yb-tserver-1

To create StatefulSets for Multi-AZ (where each has a yb-master and knows about all yb-master in the cluster) I used an override with isMultiAz: True. If you don't want the LoadBalancer, you can disable it with enableLoadbalancer: false

Those values are visible with:

helm show all yugabytedb/yugabyte
Enter fullscreen mode Exit fullscreen mode

Image description
and if you don't ind the documentation and comments sufficient, the template is: yugabyte/charts service.yaml

Note that Helm Charts are the maintained and recommended way to Install YugabyteDB on Kubernetes. There's no need for operators as the database itself is self-healing, autonomously repairing itself autonomously without the need for additional components. You scale by scaling the StatefulSets and the connections, SQL processing and data are rebalanced automatically. When a pod is down, the remaining ones do the necessary Raft Leader election to continue to serve consistent reads and writes.

Top comments (0)