Using Amazon’s Kubernetes Distribution Everywhere with Amazon EKS Distro

Using Kubernetes in the public cloud is easy. Especially when using managed services like Amazon Elastic Kubernetes Service (Amazon EKS). EKS is one of the most popular Kubernetes distros. When using EKS, there is no need to administer examination level nodes, etcd nodes or other examination level components. This simplicity allows you to focus on your application. But in real life scenarios, you sometimes need to run Kubernetes clusters in on-premises environments. Maybe because of the regulation restrictions, compliance requirements or you may need the lowest latency when accessing your clusters or applications.

There are so many Kubernetes Distributions out there. Most of them are CNCF certified. But that means you need to choose from many options, try and implement one that suits your needs. Check the security part of your distro or find a suitable tool for your deployment. In other words, we all need standardization.

In December 2020, AWS announced the distribution of EKS. EKS Distro is a Kubernetes distribution built and powered by Amazon EKS managed, allowing you to deploy secure and reliable Kubernetes clusters in any environment. EKS Distro allows you to use the same tools and the Kubernetes version and dependencies with EKS. You also don't have to worry about security patches for the distribution because with each release of the EKS distribution you also get the latest patches and the EKS distribution follows the same EKS process to check Kubernetes versions. This means that you always use reliable and tested Kubernetes distributions in your environment.

EKS Distro is an open source project on GitHub. You can check out the repository at this link. https://github.com/aws/eks-distro/

You can install EKS Distro on metal servers or virtual machines in your data center or even in the environment of other public cloud providers. Unlike EKS, when using EKS Distro, you have to manage all control level nodes, nodes, etc. and control level components yourself. This comes with some additional operational burden, but is a huge benefit if you don't have to think about the security or reliability of your Kubernetes deployment.

As you can see from the screenshot above each EKS Deployment Option has its own features. On the right column there are options and features of EKS Distro. As I’ve mentioned before when using EKS Distro you need to have your own infrastructure and you need to manage the Control Plane. Also you can use different 3rd party CNI Plugins according to your needs. Biggest difference is that unlike EKS Anywhere, there are no Enterprise Support offerings from AWS with EKS Distro.

The project is on GitHub and supported by the Community. When you have any problems or when you want to contribute to the projects you can file an issue or find solutions from the previous issues on the repository.

Let’s see EKS Distro in action!

When installing EKS Distro, you can choose a launch partner’s installation options or you can use familiar community options like kubeadm or or kops.

I will demonstrate the installation of EKS Distro with kubeadm in this blog post.

First of all, for the installation with kubeadm, you need an RPM-based Linux system. I am using a CentOS system for this demonstration.

I have installed Docker 19.03 version, disabled swap and disabled SELinux on the machine.

I will install kubelet, kubectl and kubeadm with the commands below on the machine. I will install the 1.19 version of the Kubernetes in this demonstration.

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

cd /usr/bin

sudo wget https://distro.eks.amazonaws.com/kubernetes-1-19/releases/4/artifacts/kubernetes/v1.19.8/bin/linux/amd64/kubelet; \
sudo wget https://distro.eks.amazonaws.com/kubernetes-1-19/releases/4/artifacts/kubernetes/v1.19.8/bin/linux/amd64/kubeadm; \
sudo wget https://distro.eks.amazonaws.com/kubernetes-1-19/releases/4/artifacts/kubernetes/v1.19.8/bin/linux/amd64/kubectl

sudo chmod +x kubeadm kubectl kubelet

sudo systemctl enable kubelet

After enabling kubelet service, I am adding some arguments for kubeadm.

sudo mkdir /var/lib/kubelet

sudo vi /var/lib/kubelet/kubeadm-flags.env

KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd —network-plugin=cni —pod-infra-container-image=public.ecr.aws/eks-distro/kubernetes/pause:3.2"

I will pull the necessary EKS Distro container images and tag them accordingly.

sudo docker pull public.ecr.aws/eks-distro/kubernetes/pause:v1.19.8-eks-1-19-4;\
sudo docker pull public.ecr.aws/eks-distro/coredns/coredns:v1.8.0-eks-1-19-4;\
sudo docker pull public.ecr.aws/eks-distro/etcd-io/etcd:v3.4.14-eks-1-19-4;\
sudo docker tag public.ecr.aws/eks-distro/kubernetes/pause:v1.19.8-eks-1-19-4 public.ecr.aws/eks-distro/kubernetes/pause:3.2;\
sudo docker tag public.ecr.aws/eks-distro/coredns/coredns:v1.8.0-eks-1-19-4 public.ecr.aws/eks-distro/kubernetes/coredns:1.7.0;\
sudo docker tag public.ecr.aws/eks-distro/etcd-io/etcd:v3.4.14-eks-1-19-4 public.ecr.aws/eks-distro/kubernetes/etcd:3.4.13-0

I will add some other configurations as well.

sudo vi /etc/modules-load.d/k8s.conf

br_netfilter

sudo vi /etc/sysctl.d/99-k8s.conf

net.bridge.bridge-nf-call-iptables = 1

Now let’s initialize the cluster!

sudo kubeadm init --image-repository public.ecr.aws/eks-distro/kubernetes --kubernetes-version v1.19.8-eks-1-19-4

This output is mostly the same as the usual kubeadm init command output. As you can see from the screenshot, the output has the kubeadm join command for the worker nodes or the configuration for accessing the cluster with the kubeconfig file. By the way, let me do that and access my Kubernetes cluster installed with EKS Distro.

sudo mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Let’s run kubectl get nodes command and see the output.

As you can see, I am able to connect the cluster and see the kubectl get nodes command output but node is in NotReady status. The reason is I need to add a Pod Network addon and install a CNI Plugin to the cluster. I will use Calico CNI for this demonstration.

sudo curl https://docs.projectcalico.org/manifests/calico.yaml -O
kubectl apply -f calico.yaml

After installing the Calico CNI, my master node is now in Ready state.

I have configured the worker nodes with the same prerequisites like installing Docker and disabling swap. I will pull and tag the necessary container image for the Kubernetes cluster as well with these commands.

sudo docker pull public.ecr.aws/eks-distro/kubernetes/pause:v1.19.8-eks-1-19-4;\
sudo docker tag public.ecr.aws/eks-distro/kubernetes/pause:v1.19.8-eks-1-19-4

I can now move on with adding a worker node to the cluster. I will use the kubeadm join command from the kubeadm init command output.

When I run the kubectl get nodes command, I can see the other node in Ready state.

As you can see my worker node has now joined the cluster and I can see the pods in the kube-system namespace.

My Kubernetes cluster is installed with EKS Distro and ready for deploying application workloads!

Conclusion

Having a tested, verified and reliable Kubernetes Distribution for production workloads is extremely crucial. This is why EKS is one of the most used and most popular Kubernetes distribution. Being able to run the same Distribution Amazon uses with managed EKS service on any infrastructure and platform is a huge advantage.

If you have some compliance requirements or regulation restrictions and can not use public cloud platforms you can absolutely give EKS Distro a try.