K3s Update TLS SANs

Introduction

When I was following the official guide of adding an external cluster load balancer for K3s using HAProxy to have some HAs when connecting to the cluster. I ran into a problem: kubectl refuses to connect to the cluster due to some certificate issues:

couldn't get current server API group list: Get "https://balancer_ip:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate is valid for [list of control plan ips and cluster + Kube API server service ip], ::1, not load_balancer_ip

This is due to my load balancer's IP not being in the cluster's API server certificate. During cluster initialization and when adding new control plane node to the cluster, K3s automatically update this certificate to include the new control plane node. However, this is not the case when adding an external cluster load balancer.

Table of Contents

• Solution 1: Reinitialize the Cluster With --tls-san
• Solution 2: Editing K3s Configuration File
  • Step 1: Edit the Config File
  • Step 2: Delete the Existing Kube API Server Cert
  • Step 3: Restart K3s
  • Step 4: Inform the DynamicListener About the Change
    • Alternative Method
• Worker / Control Plane Node Reconfiguration (Optional)

Solution 1: Reinitialize the Cluster With --tls-san

According to the K3s official documentation, you can pass the --tls-san flag when initializing the cluster:

sh k3s_install.sh --tls-san additional_ip_1 --tls-san additional_ip_1 --cluster-init

However, my cluster is already running and I don't want to reinstall it.

Solution 2: Editing K3s Configuration File

The K3s configuration file is located under /etc/rancher/k3s/config.yaml on all control plane nodes. If the file doesn't exist, you have to create it manually.

Step 1: Edit the Config File

In the config file, you can simply add the following entries:

tls-san:
  - ip_1
  - hostname_1
  // additional entries

This can be done on any control plane node.

Step 2: Delete the Existing Kube API Server Cert

On any control plane node run

sudo kubectl -n kube-system delete secrets/k3s-serving

to delete the existing Kubernetes API server certificate.

This can be done anywhere as long as you can access the cluster's API server.

Step 3: Restart K3s

Restart K3s on the same control plane node:

sudo systemctl restart k3s

Step 4: Inform the DynamicListener About the Change

DynamicListener is a component of K3s that handles automatic updates/renewal of the API server certificate, including when new control plan nodes join the cluster.

According to one of the K3s contributor, Brad Davidson in one of the GitHub issue:

Dynamiclistener adds SANs for any hostname or IP address requested via a HTTP Host header or TLS SNI handshake. It is designed to allow you to add or remove servers without having to manually regenerate the certificate. The downside (as you noted) is that this allows for SAN stuffing.

The documentation about it is not very clear and the official README is still in progress. This is as far as I can go about it.

For each new hostname / IP you added to the config file, run the following command on the control plan node:

curl -k --resolve your_new_hostname_or_ip:6443:127.0.0.1  https://your_new_hostname_or_ip:6443/ping

This method is also mentioned in another GitHub issue comment here.

To confirm that the API server certificate has been updated, run the following command:

kubectl get secret/k3s-serving -n kube-system -o yaml

This can be done anywhere as long as you have access to the cluster's API server.

In the metadata.annotations part of the output, you should see your newly added hostname/ip as one of the annotations:

listener.cattle.io/cn-ur_hostnameor__ip: ur_hostname_or_ip

Alternative Method

I found one Medium post here that uses another method to update the DynamicListener.

After you delete the existing API server certificate on a control plane node using

sudo kubectl -n kube-system delete secrets/k3s-serving

You can move or delete a file called dynamic-cert.json located under /var/lib/rancher/k3s/server/tls on the same control plane node

sudo mv /var/lib/rancher/k3s/server/tls/dynamic-cert.json /tmp/dynamic-cert.json

BEFORE restarting K3s on the same control plane node.

I have tested both methods and they all worked.

Worker / Control Plane Node Reconfiguration (Optional)

Now we have to update our worker / control plane nodes to use the new endpoint.

The fastest way is to update the K3s systemd service file.

You will have to do this on each node that you want to reconfigure.

For control plane nodes the file is located at

/etc/systemd/system/k3s.service

For worker nodes the file is located at

/etc/systemd/system/k3s-agent.service

Update the value of the --server flag in the file to use the new hostname / IP of the balancer.

After making the changes, run sudo systemctl daemon-reload to reload the changes.

Finally run sudo systemctl restart k3s to restart the K3s daemon for the changes to take effect.