I recently found an interest in Kubernetes and learned about it at night, while working at something not web-related at day. As part of my learning journey I wanted to quickly see and experience how Kubernetes actually works in action. So I decided to write a few services that can be used to trigger and observe certain behavior of Kubernetes. I started with Load Balancing, Self Healing of Services and Auto Scaling depending on CPU utilization.

In this blog post I will explain how each service works and how Kubernetes behaves in practice. Note that this was the first time I wrote Go, so I take no guarantee that the code is not shitty or against Go conventions and best practices that I do not yet know. :-)

This blog post addresses anyone who already has a Kubernetes cluster up and running. You should know what a Pod is, a Replica Set or a Deployment, and can build Docker containers and have used a Docker Registry.

If you do not run a Kubernetes cluster already, I can recommend the book "Kubernetes: Up & Running" and/or the blog post "How to Build a Kubernetes Cluster with ARM Raspberry Pi" by Scott Hanselman. If you don't know what Kubernetes is or how it works, you can read the excellent series of blog posts "Understanding Basic Kubernetes Concepts" by Puja Abbassi from giantswarm.io.

For the impatient

Before running kube-alive on your cluster, make sure that you have the following:

• kubectl installed and configured to a running cluster (check that "kubectl get nodes" gives you a list of at least one node in state "Ready")
• bash
• Your cluster runs on Linux on amd64 or ARM CPUs

If you do not have a cluster up and running already, I recommend the already mentioned article by Scott Hanselman to start a cluster on Raspberry Pis, or you can use Minikube to run a local cluster on your PC or Mac.

If you just want to deploy kube-alive to your cluster and see it in action, you can do this with this single command:

curl -sSL https://raw.githubusercontent.com/daniel-kun/kube-alive/master/deploy.sh | bash

Using 192.168.178.79 as the exposed IP to access kube-alive.
deployment "getip-deployment" created
service "getip" created
deployment "healthcheck-deployment" created
service "healthcheck" created
deployment "cpuhog-deployment" created
service "cpuhog" created
horizontalpodautoscaler "cpuhog-hpa" created
deployment "frontend-deployment" created
service "frontend" created

FINISHED!
You should now be able to access kube-alive at http://192.168.178.79/.

Load-Balancing

The most basic capability of Kubernetes is load balancing between multiple services of the same kind. To observe whether the request was served from the same or from different instances, I decided to let the service return it's host's IP address. In order to run a service in Kubernetes, you need to a) write the service, b) build a container hosting the service, c) push the container to a registry, d) create an object in Kubernetes that runs your container and finally e) make the service accessible from outside the cluster.

Phase A: Writing the Service

So let's dig into the code. I wrote a server in Go that serves on port 8080, parses the output of the command "ip a" and returns the container's IP address.

package main
import "fmt"
import "bufio"
import "os/exec"
import "log"
import "strings"
import "net/http"

/**
getip starts an HTTP server on 8080 that returns nothing but this container's IP address (the last one outputted by "ip a").
**/
func getIP() string {
    // Left out for brevity, see 
https://raw.githubusercontent.com/daniel-kun/kube-alive/master/src/getip/main.go 
}

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, getIP())
    })
    fmt.Printf("'getip' server starting, listening to 8080 on all interfaces.\n")
    http.ListenAndServe(":8080", nil)
}

Phase B: Building the Container

Since everything running in Kubernetes must be a container, I wrote a Dockerfile to run this service:

FROM golang
COPY main.go /go/src/getip/main.go
RUN go install getip
ENTRYPOINT /go/bin/getip

This Dockerfile is simple: It uses a golang base container that is prepared to compile and run Go code. It then copies over the only source code file, main.go and compiles and installs it to /go/bin/ using "go install".

The installed binary /go/bin/getip is set as the Entrypoint, so that when no argument is given to docker run, it executes our service.

You can build the container using:

docker build .

Note that there is a "." at the end of the command, meaning that you must have cd'ed to the getip source directory before executing docker build.

After docker build finishes, you will be able to see the new container with a new, randomly generated image id via

docker images

The container is only available locally on the machine that it has been built. Since Kubernetes will run this container on any node that it sees fit, the container must be made available on all nodes. That's where a Docker Registry steps into the game, which is basically a remote repository for Docker containers that is available from all nodes.

Phase C: Pushing the Container to a Registry

I first tried to set up a local registry, which can be done, but the setup is not
portable across clusters. That's why I decided to simply use Docker's own registry, https://hub.docker.com. To push your freshly built container, you first need to register at Docker Hub, then tag the container with the repository, desired container name and an optional tag. If no tag is given, "latest" is assumed.

docker tag <your-repository>/getip <image id> # tag the docker image with your repository name and the service name, such as "getip"
docker login # enter your username and password of http://hub.docker.com now.
docker push <your-repository>/getip # and then push your container

It is now available to be pulled (without authorization) by anyone - including your Kubernetes nodes.

Phase D: Define a Replica Set

To let Kubernetes know that it should run this container as a service, and to run multiple instances of this service, you should use a Replica Set. I wrapped the Replica Set into a Deployment to easily upgrade the service later:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: getip-deployment
  labels:
    app: getip
spec:
  replicas: 4
  selector:
    matchLabels:
      app: getip
  template:
    metadata:
      labels:
        app: getip
    spec:
      containers:
      - name: getip
        image: <your-repository>/getip
        ports:
        - containerPort: 8080

I set the number of replicas to 4, which means that Kubernetes will do everything it can to always have exactly 4 instances running at any time. However, this does not give us a single URL to connect to these instances. We will use a Service to Load Balance between these instances:

kind: Service
apiVersion: v1
metadata:
  name: getip
spec:
  selector:
    app: getip
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

This service provides a single, load-balanced URL to access the individual service instances. It remaps default HTTP port 80 to the service's own port 8080 in the process. The service will be available as http://getip.svc.default.cluster or, even shorter, as http://getip on any running Kubernetes Pod.

However, this service is only available from inside Kubernetes and not from "outside" the cluster.

Phase E: Publish the Service

I decided to build my own nginx container to serve the static HTML and JavaScript files that make up the frontend and publish the services from a specific IP.

events {
    # empty
}
http {
    server {
        root /www/data;
        location / {
            # for the frontend SPA
        }
        # Forward traffic to <yourip>/getip to the getip service.
        location /getip {
              proxy_pass http://getip/;
        }
        # I have left out the other services like "cpuhog" and "healthcheck" here for brevity.
        # See their code on https://github.com/daniel-kun/kube-alive/

        # Allow WebSocket connections to the Kubernetes API:
        location /api {
              proxy_pass https://kubernetes.default/api;
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection "Upgrade";
              proxy_set_header Authorization "Bearer %%SERVICE_ACCOUNT_TOKEN%%";
        }
    }
}

So we see that nginx expects the SPA in /www/data/, which will be the target of COPY commands in our Dockerfile. The service getip is reached via Kubernetes DNS, which will automatically resolve a service's name to it's Cluster IP, which in turn load-balances requests to the service instances. The third location /api is used by the frontend to receive information about running pods. (Currently, the full API is exposed with full admin privileges, so this is highly insecure - do it in isolated environments only! I will fix this in the near future.)

Here's the Dockerfile for the frontend service:

FROM nginx
COPY nginx.conf /etc/nginx/
COPY index.html /www/data/
COPY output/main.js /www/data/output/main.js
COPY run_nginx_with_service_account.sh /kube-alive/
CMD /kube-alive/run_nginx_with_service_account.sh 

The shell script run_nginx_with_service_account.sh will substitute variables in the nginx.conf to use the Kubernetes Service Account token in the authorization header to let nginx handle the authorization so that the frontend does not have to.

So now we are prepared to put the last piece of the puzzle into place: A Replica Set to run the frontend and a Service that externally publishes the frontend. Note that I wrapped the Replica Set into a Deployment again:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: frontend-deployment
  labels:
    app: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        imagePullPolicy: Always
        image: <your-repository>/frontend_amd64
        ports:
        - containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
  name: frontend
spec:
  selector:
    app: frontend
  externalIPs:
  - <put your external IP, e.g. of your cluster's master, here>
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

That's it! You can kubectl apply this after you inserted a valid externalIP and everything should be up and running to execute your first experiment with Kubernete's load balancing.

Reaching to the IP that your "kubectl" is configured against, should give you this UI:

… followed by more experiments. These all follow the same concept as getip, you can have a look at their code and deployment yamls here:

Self-Healing

The code of the Self-Healing experiment is here:

https://github.com/daniel-kun/kube-alive/tree/master/src/healthcheck
https://github.com/daniel-kun/kube-alive/blob/master/deploy/healthcheck.yml

Rolling Updates

The code of the Rolling Updates experiment is here:

https://github.com/daniel-kun/kube-alive/tree/master/src/incver
https://github.com/daniel-kun/kube-alive/blob/master/deploy/incver.yml

Auto Scaling (cpu-based)

The code of the Auto Scaling experiment is here:

https://github.com/daniel-kun/kube-alive/tree/master/src/cpuhog
https://github.com/daniel-kun/kube-alive/blob/master/deploy/cpuhog.yml

I named the service "cpuhog" because it uses as much CPU as it can for 2 seconds for every request.

I plan to add more experiments in the future, such as an experiment for rolling updates using Deployments.

I hope that you found this blog post and the kube-alive services useful and would be thankful if you could leave feedback in the comments. Maybe one day kube-alive will be a starting point to see Kubernetes behaviour live in action for many starters and engineers that are evaluating Kubernetes for their own use.

Update from 01/25/2018: I removed the security warning, because security on the latest version of kube-alive has been tightened. Only a portion of the API is exposed (pods in the kube-alive namespace), and the frontend runs with a service account that has access only to the dedicated kube-alive namespace and only to read and list Pods. Hence, there is not much more information available via the API than is visible in the frontend anyways.

Update from 03/08/2018: Updated the gifs to the new, improved visuals and added the rolling updates experiment.