Kubernetes: Dynamic Admission Control

This is a demo for Kuberentes Dynamic Admission Control, exploring various concepts, capabilities and constraints, it's following A Guide to Kubernetes Admission Controllers which contains a lot of information around Admission Controllers and Kubernetes.

What is Dynamic Admission Control

Borrowing the definion from the Kubernetes page

In addition to compiled-in admission plugins, admission plugins can be developed as extensions and run as webhooks configured at runtime.

Admission webhooks are HTTP callbacks that receive admission requests and do something with them. You can define two types of admission webhooks, validating admission webhook and mutating admission webhook.

You can read some of the use cases for admission controllers.

In the demo we will focus on validating admission webhooks, but most of the principles apply on mutating as well.

Demo

We are going to run the demo locally on minikube.

Prerequisites

You will need some tools installed for following along

• Clone code from Github
• minikube
• kubectl
• docker
• go (optional if you want to play with webhook server code)

Setting up minikube

We are going to spin up a minikube cluster by running

# start kubernetes cluster locally
minikube start

# enable needed addons
minikube addons enable metrics-server
minikube addons enable registry

We are setting up a local image registry so we don't rely on pushing/pulling the webhook server container image on a remote registry e.g. docker hub.

# run local registry
docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000"

All the components for running and testing the webhook now should be ready.

Building and Deploying

The next step is to build the Webhook Docker image and deploy it alongside with the Admission Webhook configuration.

1. You will need to build and push the container to your local registry by running:

make docker-image-local

1. then you are ready to deploy the Validating Admision Webhook in your local minikube cluster with:

make deploy

The make deploy command does a couple of actions

1. Generates TLS Certificates, since a webhook must be served via HTTPS, we need proper certificates for the server. [ More ]
2. Creates a dedicated namespace for putting the Webhook Server Deployment and the ValidatingWebhookConfiguration. This useful if you want to skip your Webhook from "validating itself".
3. Creates a Kubernetes TLS secret with the generated key and certificate so the Webhook server can access.
4. Creates a Service for reaching the Webhook Server Pods at port 443.
5. Creates the Webhook Server Deployment.
6. Finally after everything is in place creates the ValidatingWebhookConfiguration.

The Webhook Server

The Webhook Server we are deploying is a simple Go HTTP server that accepts AdmissionReview POST requests at /validate, verifys the payload is correct.

Then checks the resource passed for review, in the demo case we are only care about POD creation, if it contains a team label set up. You can configure rules for what review requests are directed to your Webhook in ValidatingWebhookConfiguration object. More details on Matching requests: rules.

If this label team: <value> is missing the Webhook Server will return either a warning that the lable is missing or a failure that prevents the POD from scheduling depending on the ALLOW_SCHEDULING env var.

if _, exists := pod.Labels["team"]; !exists {
    admissionReviewResponse.Response.Allowed = allowScheduling
    if allowScheduling {
        admissionReviewResponse.Response.Warnings = []string{
            "Team label not set on pod",
        }
    }
    admissionReviewResponse.Response.Result = &metav1.Status{
        Status:  "Failure",
        Message: "Team label not set on pod",
    }

    log.Printf("Team label not set on pod: %s\n", pod.Name)
}

In the warning case you will see this message

Warning: Team label not set on pod

Also it's a good practise to skip validating namespaces that are used by Kubernetes, e.g. kube-public and kube-system, as you might cause a disruption of you cluster if you deny scheduling of resources there.

You can experiment with the Webhook Server code do changes and redeploy it. You can build a new docker image and push it to the local registry we set up earlier with

make docker-image-local

and then restart the deployment for creating new PODS with the new image

kubectl rollout restart deploy/webhook-server

Verifying everything Works

Okay let's check that everything works and see the Validating Webhook in practise.

First we can check that the Webhook Server is scheduled and healty. You can do it with

kubectl get pods -n webhook-demo

and the output should be similar to this

NAME                              READY   STATUS    RESTARTS   AGE
webhook-server-79c79b5877-d624n   1/1     Running   0          11m
webhook-server-79c79b5877-dpf4b   1/1     Running   0          11m

Then if we try to schedule a simple POD it should fail to schedule with the error

kubectl apply -f deployment/busy-box.yaml

Error from server: error when creating "deployment/busy-box.yaml": admission webhook "webhook-server.webhook-demo.svc" denied the request: Team label not set on pod

You can add a team label in the pod spec and try again successfully.

Observations

Namespace Selector

You might have noticed the code in the Webhook Server contains logic for skipping validation for a list of namespaces.

But ValidatingWebhookConfiguration supports natively limiting requests that reach to your Webhook Server depending on namespaceSelector.

I couldn't find a way to whitelist namespaces by their name. For using namespaceSelector you can add this to ValidatingWebhookConfiguration.

namespaceSelector:
  matchExpressions:
    - key: webhook-skip
      operator: DoesNotExist

and then mark all namespaces you want to skip with this label

kubectl label ns webhook-demo webhook-skip=true

Reliability and availability

Dynamic Admission Controller can be on the critical path, depending how you your Webhook configured e.g. can block various actions on Kubernetes resources.

This is a non exhaustive list of things you can do to increase the reliability and the availability of your admission controller.

• The Dynamic Admission Controller is backed by an HTTP Server and a Deployment. This means all concepts for making a stateless deployment more reliable in Kubernetes ( if it is hosted in Kubernetes )
  • Use load balancer in front of your Webhook server that provides High Availability
  • Use Rolling Updates
  • Pod Disruption Budget
  • Rollback to an earlier Deployment if the current state is not stable
  • Autoscalling for matching increasing demand [ HPA ]
• Avoiding deadlock in self hosted webhooks
• Avoiding operating on the kube-system namespace
• It is recommended that admission webhooks should evaluate as quickly as possible (typically in milliseconds), since they add to API request latency. It is encouraged to use a small timeout for webhooks.
• Setting Failure Policy depending on your needs.
• And last but not least always monitoring and observing your Webhooh Server status and the Kube API server status. The API Server exposes Prometheus metrics from the /metrics endpoint e.g. apiserver_admission_webhook_admission_duration_seconds_sum,apiserver_admission_webhook_rejection_count.

Summary

Dynamic Admission Controllers provide great benefits for security and governance across your Kubernetes Cluster. This demo can help you get started building and deploying a Validating Admission Webhook.

Resources

• Dynamic Admission Control
• A Guide to Kubernetes Admission Controllers
• Github repository