DataOps: Kubernetes on local with vagrant and kubeadm

This series is a story about a data guy (which is me) with zero knowledge of Kubernetes pick up devops skill and successfully deploy a machine learning model.

This series is a story about a data guy (which is me) with zero knowledge of Kubernetes pick up devops skill and successfully deploy a machine learning model.

I have been working with Kubernetes for a few months, the cluster is set up by devops team and me myself is only a user. It’s time for me to take a break, build my own local cluster and understand deeply the whole process.

If you guys don’t want to setup a k8s cluster from scratch, Microk8s is the best choice, you can get a full kubernetes system running in under 60s (like snap a finger)

Kubeadm vs Minikube.

After researching, I found two candidates that good for my local cluster:

Minikube is the most popular way to deploy Kubernetes locally, it comes with support for a variety of hypervisors, set up is completely automatic within a few minutes. You really doesn’t need to take care anything.

However, I want to get my hand dirty by doing some installation stuffs as well as understand how to set up Kubernetes cluster. Hence, one of the most frequency criticism that Kubernetes is very hard to install, Kubeadm really makes this easier.

Installation

TL;DR: There are 2 the basic steps that I did:

• Spin up a vagrant machine with Ubuntu 16.04 and at least 2GB of memory.
• Install and run kubeadm on single master cluster. (official guide can be found at https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)

Vagrant

A simple vagrant with Ubuntu 16.04 and 2Gb of memory is good enough for this series. You can copy the following code, put it to Vagrantfile and run it with vagrant up .

Vagrant.configure("2") do |config|
  config.vm.box = "bento/ubuntu-16.04"
  config.vm.box_check_update = false
  config.vm.network "private_network", type: "dhcp"
  config.vm.provider "virtualbox" do |vb|
    vb.gui = false
    vb.memory = "2048"
    vb.customize ["modifyvm", :id, "--cableconnected1", "on"]
  end
  config.vm.provision "docker"
end

Inside the VM, I do a couple of prerequisite steps. Turn off swap and make sure it won’t run again whenever I reboot my machine.

$ swapoff -a 
$ sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Kubeadm

I follow the official guide to install kubernetes on ubuntu.

$ apt-get update && apt-get install -y apt-transport-https curl
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
$ cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
$ apt-get update
$ apt-get install -y kubelet kubeadm kubectl
$ apt-mark hold kubelet kubeadm kubectl

The most difficult for me during the installation is to config the cgroup driver used by Kubelet. Luckily, A post from Liz Rice saves me, she did the kubeadm on vagrant with a little hack code, I copy it from her.

$ sed -i '0,/ExecStart=/s//Environment="KUBELET_EXTRA_ARGS=--cgroup-driver=cgroupfs"\n&/' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Initializing the master with IP which generated from DHCP, the reason for it is we can connect to our cluster from outside/host machine.

$ kubeadm init --apiserver-cert-extra-sans=<IP Address>  --node-name $(hostname -s)

To get IP Address, we use ifconfig command, the default network name is eth1

$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 08:00:27:1c:68:af
          inet addr:172.28.128.4  Bcast:172.28.128.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe1c:68af/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2800 (2.8 KB)  TX bytes:2978 (2.9 KB)
          Interrupt:16 Base address:0xd240

I make the kubectl works for vagrant user

$ sudo --user=vagrant mkdir -p /home/vagrant/.kube
$ cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
$ chown $(id -u vagrant):$(id -g vagrant) /home/vagrant/.kube/config

Installing a pod network add-on, we can find the list from third-party pod network providers here. I choose weave net because they support multiple architect (amd64, arm, arm64 and ppc641e)

$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Finally, I make the master isolate. By default, the cluster will not schedule pods on the master for security reason and we have only one node for both master / worker.

$ kubectl taint nodes --all node-role.kubernetes.io/master-

Test our cluster is running with kubectl command.

$ kubectl get nodes
NAME      STATUS   ROLES    AGE     VERSION
vagrant   Ready    master   3h28m   v1.12.1

Below is my sh file for install kubeadm. You also can find the Vagrantfile with full installation process at my repo: https://github.com/canhtran/data-devops-medium/tree/master/vagrant.d

# Install kubernetes
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

# kubelet requires swap off
swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# Configure cgroup driver used by kubelet on Master Node
sed -i '0,/ExecStart=/s//Environment="KUBELET_EXTRA_ARGS=--cgroup-driver=cgroupfs"\n&/' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

IPADDRESS=`ifconfig eth1 | grep "inet addr" | cut -d ':' -f 2 | cut -d ' ' -f 1`

# Set up Kubernetes
kubeadm init --apiserver-cert-extra-sans=$IPADDRESS  --node-name $(hostname -s)

# Make kubectl work for your non-root user
sudo --user=vagrant mkdir -p /home/vagrant/.kube
cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
chown $(id -u vagrant):$(id -g vagrant) /home/vagrant/.kube/config

# Installing a pod network add-on
export KUBECONFIG=~vagrant/.kube/config
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

# Master Isolation
kubectl taint nodes --all node-role.kubernetes.io/master-

Optional

In order to get a kubectl on other computers (e.g. laptop) talk to the cluster, we need to copy the administrator kubeconfig file from master to our workstation.

By default, the SSH access in vagrant is not enable for root. We have to copy the admin.conf file to be accessible by some users.

$ vagrant ssh # Login to vagrant
$ sudo -i # Get root access

The from our machine, we can copy it via scp and test with get nodes command. (We have to change the server ip in admin.conf with the vagrant ip which is eth1 ip).

$ scp -r -P 2222 vagrant@127.0.0.1:/home/vagrant/admin.conf .
##### Edit and change the IP address ####
$ kubectl --kubeconfig ./admin.conf get nodes
NAME      STATUS   ROLES    AGE     VERSION
vagrant   Ready    master   3h28m   v1.12.1

After we have a running cluster, let’s start with helm chart

Helm chart

Helm chart is Kubernetes package management.

Kubernetes can become very complex with a tons of objects that we need to handle: configmaps, services, pods, persistance volumnes. I should mention that the number of release of each service also a big headache for us to manage. These can be solved with Helm, which offers us a simple way to manage, deploy in one simple application with standard template (chart).

Base on the documentation

Helm uses a packaging format called charts. A chart is a collection of files that describe a related set of Kubernetes resources. A single chart might be used to deploy something simple, like a memcached pod, or something complex, like a full web app stack with HTTP servers, databases, caches, and so on.

Charts are created as files laid out in a particular directory tree, then they can be packaged into versioned archives to be deployed

Helm chart can be installed on a specific namespace. In this series, I will create a new namespace call data

$ kubectl create namespace data

Then I can install helm with simple command

$ helm init --tiller-namespace data

Helm will create a Tiller pod in Kubernetes cluster under namespace data. To verify the success of our installation.

$ kubectl get pods -n data 
NAME                             READY     STATUS    RESTARTS   AGE
tiller-deploy-5b9757b87c-s295g   1/1       Running   0          23s

That’s all, everything is ready for the next chapter.