Virtual Kubernetes Clusters: What Are They Needed For?

Developer Wishlist Never Ends

Imagine you can

• have many virtual clusters within a single cluster, and
• they are much cheaper than the traditional Kubernetes clusters, and
• they require lower management and maintenance efforts.

Sounds intriguing, eh? This makes v/clusters ideal for running experiments, continuous integration, and setting up sandbox 🧪 environments.

So, Loft Labs created such a solution written natively in Golang and made it an ~2k⭐ open source.

What's under the hood?

On top of other Kubernetes clusters, virtual clusters are fully functional Kubernetes clusters. Virtual clusters utilize the worker nodes and networking of the host cluster, as opposed to completely distinct "real" clusters. They schedule all workloads into a single namespace of the host cluster and have their own control plane. Virtual clusters divide a single physical cluster into several distinct ones, much like virtual machines do.

🖼️ Right click, don't even think too long.

Only the essential Kubernetes components - the API server, controller manager, storage backend (such as etcd, sqlite, mysql, etc.), and - optionally - a scheduler—make up the virtual cluster itself. In order to minimize virtual cluster overhead, vcluster builds by default on k3s, a fully functional, certified, lightweight Kubernetes distribution that compiles the Kubernetes components into a single binary and disables by default all unnecessary Kubernetes features, such as the pod scheduler or specific controllers.

Other Kubernetes distributions, such k0s and vanilla k8s, are supported in addition to k3s. In addition to the control plane, the virtual cluster also includes a Kubernetes hypervisor that simulates networking and worker nodes. Between the virtual and host clusters, this component syncs a few key resources that are crucial for cluster functionality:

• Pods: All the virtual cluster's started pods are rewritten before being launched in the virtual cluster's namespace in the host cluster. Environment variables, DNS, service account tokens, and other configurations are updated to point to the virtual cluster rather than the host cluster. In the pod, it appears that the virtual cluster rather than the host cluster is where the pod is started.
• Services: On the namespace of the virtual cluster in the host cluster, all services and endpoints are rewritten and generated. The service cluster IPs are shared by the host cluster and virtual cluster. This implies that there are no performance consequences when a service in the host cluster is accessed from within the virtual cluster.
• PersistentVolumeClaims: In the event that persistent volume claims are generated in the virtual cluster, they will be modified and generated in the host cluster's namespace. The relevant persistent volume data will be synchronized back to the virtual cluster if they are bound in the host cluster.
• ConfigMaps & Secrets: Only ConfigMaps and secrets mounted to pods within the virtual cluster will be synced to the host cluster; all other ConfigMaps and secrets will only be retained within the virtual cluster.
• Other Resources: Deployments, StatefulSets, CRDs, service accounts, etc. do not sync with the host cluster; instead, they only reside in the virtual cluster.

Who lost the magic mirror?

For each pod with the spec.nodeName value it encounters inside the virtual cluster, vcluster by default creates a false node. Because vcluster does not by default have RBAC permissions to access the real nodes in the host cluster because doing so would require a cluster role and cluster role binding, those false nodes are produced. Additionally, each node will get a false kubelet endpoint that will either send requests to the real node or rewrite them to keep virtual cluster names intact.

Vcluster supports multiple modes to customize node syncing behavior. For a detailed list of the resources that may have been synced, see details here in the docs.

The hypervisor also proxies some Kubernetes API calls, including pod port forwarding or container command execution, to the host cluster in addition to synchronizing virtual and host cluster resources. It essentially performs the function of the virtual cluster's reverse proxy.

To ensure proper network operation for the virtual cluster, resources like Service and Ingress are synced by default from the virtual cluster [down] to the host cluster.

There are never too many levels of abstraction

Certain resources (such as CRDs or RBAC policies) reside cluster-wide, and you can’t isolate them using namespaces. For instance, it is not feasible to install an operator simultaneously in multiple versions inside a same cluster.

$ kubectl api-resources --namespaced=false|true

Although Kubernetes itself already offers namespaces for various settings, their use of cluster-scoped resources and the control plane is constrained.

In many circumstances, virtual clusters are also more stable than namespaces. In its own data store, the virtual cluster produces its own Kubernetes resource objects. These resources are unknown to the host cluster.

This kind of isolation is good for resilience. The necessity for access to cluster-scoped resources like cluster roles, shared CRDs, or persistent volumes still exists for engineers who adopt namespace-based isolation. Each team that depends on one of these shared resources will probably experience failure if an engineer destroys something in it.

Finally, virtual cluster configuration is independent of physical cluster configuration. This is excellent for multi-tenancy because it allows you to easily create a fresh environment or amazing demo applications. 😎

How it looks in your CLI

Create file vcluster.yaml:

vcluster:
  image: rancher/k3s:v1.23.5-k3s1   # Choose k3s version

Then, install helm chart using vcluster.yaml for chart values:

helm upgrade --install my-vcluster vcluster \
  --values vcluster.yaml \
  --repo https://charts.loft.sh \
  --namespace host-namespace-1 \
  --repository-config=''

Access:

Get the admin tool

curl -s -L "https://github.com/loft-sh/vcluster/releases/latest" | sed -nE 's!.*"([^"]*vcluster-linux-amd64)".*!https://github.com\1!p' | xargs -n 1 curl -L -o vcluster && chmod +x vcluster;
sudo mv vcluster /usr/local/bin;

Then, connect:

# Connect and switch the current context to the vcluster
vcluster connect my-vcluster -n my-vcluster

# Switch back context
vcluster disconnect

You have an option to create a separate kubeconfig to use instead of changing the current context:

vcluster connect my-vcluster --update-current=false

Or you may execute a command directly with vcluster context without changing the current context:

vcluster connect my-vcluster -- kubectl get namespaces
vcluster connect my-vcluster -- bash

Usage:

# Run any kubectl, helm, etc. command in your vcluster
kubectl get namespace
kubectl get pods -n kube-system
kubectl create namespace demo-nginx
kubectl create deployment nginx-deployment -n demo-nginx --image=nginx
kubectl get pods -n demo-nginx

Cleanup:

helm delete my-vcluster -n vcluster-my-vcluster --repository-config=''

What if you're planning some serious thing?

Well, the stock K8s distribution is compatible with high availability in vcluster. What is meant by high availability? Well, one of the entities is to make etcd database more robust. The second one is to boost syncer's performance. As mentioned above, vcluster uses a so-called syncer which copies the pods that are created within the virtual cluster to the underlying host cluster.

🪲TL;DR #1: etcd uses a leader-based consensus protocol for consistent data replication and log execution. Etcd cluster members elect a single leader, all other members become followers. The cluster elects a new leader automatically when one falls out of favor. Once the incumbent fails, the election does not take place immediately. Since the failure detection methodology is timeout based, electing a new leader takes roughly an election timeout.

🪲TL;DR #2: Why it is recommended to have a minimum of three instances in an etcd cluster, is well described here, a first hand info.

Currently, vcluster's high availability setup does not allow single binary distributions like k0s and k3s.

Create a values.yaml with the following structure in order to operate vcluster in high availability mode:

# Enable HA mode
enableHA: true

# Scale up syncer replicas
syncer:
  replicas: 3

# Scale up etcd
etcd:
  replicas: 3

# Scale up controller manager
controller:
  replicas: 3

# Scale up api server
api:
  replicas: 3

# Scale up DNS server
coredns:
  replicas: 3

To summarize

A virtual Kubernetes cluster that is fully functioning can be built using vcluster! The underlying K8s cluster's namespace is where each vcluster operates. It provides better multi-tenancy and isolation than conventional namespaces, and it is less expensive than building independent, fully-fledged clusters.

Virtual clusters can be a good option to running numerous instances of k3s, or k0s side by side, but they cannot exist on their own without a host cluster.

Compared to fully independent Kubernetes clusters, they are faster, lighter, and simpler to reach. Therefore, give using a virtual cluster a shot if you're tired of having to reset your local or CI/CD Kubernetes clusters all the time. However, this is a topic for a completely different story, much sadder than what you just read.

Be in good & non-ghost shape! 👻

Many thanks to Viktor 🐦@vfarcic Farcic and Mauricio 🐦@salaboy Salatino for inspiration!