Barbara

Posted on Oct 9, 2023

Kubernetes Design

#desgin #kubernetes #resourcemanagement #framework

In this blogpost, you will get a crisp guide through the design concepts of Kubernetes. Let's go:

Decoupled resources

Each component should be decoupled from outer resources, so that every component can be removed, replaced or rebuilt.
Use Services for connections to other resources to provide flexibility.

Transience

Each object should be built with the expectation that other components will die and be rebuilt.
Having this in mind, we can update and scale with ease.

Flexible Framework

Multiple independent resources work together, but they are decoupled and do not expect a permanent relationship to other resources.
This framework of independent resources is not as efficient, as we have a lot of controllers or watch-loops in place to monitor the current cluster state and change things until the state matches the configuration.
But on the other hand, this framework allows us to have more flexibility, a very high availability and scalability.

Resource Usage

Kubernetes allows us to easily scale clusters and sets resource limits via configuration.

CPU

spec.containers[].resources.limits.cpu
spec.containers[].resources.requests.cpu

1 CPU in K8s is equivalent to 1 AWS vCPU or 1GCP Core or 1 AzurevCore or 1 Hyperthread on a bare-metal Intel processor with Hyperthreading (eg to behave like multiple virtual cores).

RAM

spec.containers[].resources.limits.memory
spec.containers[].resources.requests.memory

With Docker the limits.memory value is converted to an integer value to be used in the docker run --memory <value> <image> command.
If the container exceeds its memory limit, it may be restarted or the entire Pod could be evicted from the node.

Ephemeral Storage

Container files, logs can be stored there. If the containers use more than the limit in the Pod, the Pod will be evicted.

spec.containers[].resources.limits.ephemeral-storage
spec.containers[].resources.requests.ephemeral-storage

Label Selectors

They provide a flexible and dynamic way to interact with your K8s cluster and help with the following points:

Resource Organization
Resource Identification
Selective Resource Access
Application Deployment
Scaling and Load Balancing
Rolling Updates and Rollbacks
Monitoring and Logging
Multi-Tenancy
Custom Workflows

Selectors are namespace scoped, you can add the --all-namespaces argument to select matching objects in all namespaces

kubectl get object-name -o yaml
kubectl get pod pod-name -o yaml

labels:
  app: sample
  pod-template-hash: 0815

kubectl -n yourns get --selector app=your_pod

Multi-Container Pods

Having multiple containers allows independent development and scaling for every container to best meet the needs of the workload.
Every container in a POD shares a single IP address and namespace.
Each container has equal potential access to storage given to the Pod.

Different types of containers

Ambassador

Used to communicate outside resources, often outside a cluster. With this you don't need to implement a new service or a new entry to an ingress controller.

Adapter

is used to modify the data generated by the primary container. An example would be a datastream that needs to be modified for a usecase.

Sidecar

you can compare it to a sidecar on a motorcycle. It often provides services that are not found in the main application. For example a logging container. So it remains decoupled and scalable.

initContainer

An init container allows one or more containers to run only if one or more previous containers run and exit successfully.
For example a git-sync container would be an init container for another applications that needs to have always the latest information from a given git.

spec:
  containers:
   -name: app
    image: app-image
  initContainers:
  - name: git-sync
    image: git-sync-image

CRD - Custom Resource Definition

With CRDs you can extend the K8s API and create custom resource.
With the help of a CRD you can add databases, message queues or machine learning models and many more and create custom schemas for your custom resource.
There are also public CRDs that can be used. For example Helm CRDs or the Prometheus Operator CRDs.

Job

It is a resource object to manage and run a task or a batch process. Jobs ensure that a given number of pods successfully completes.

Characteristics

One-Time Execution. For example a database migration or a backup
Parallelism. number of parallel pod completions
Pod Template. defines the container(s) to run
Completion and Failure Handling. can be defined via completions and backoffLimit. The backoffLimit defines the number of retries.
Garbage collection.

Sample of a job manifest.yaml in K8s

apiVersion: batch/v1
kind: Job
metadata:
  name: your-job
spec:
  completions: 3  # Number of desired completions
  parallelism: 1  # Number of pods running in parallel
  template:
    spec:
      containers:
      - name: your-container
        image: your-image
        command: ["echo", "Hello World!"]
  backoffLimit: 2  # Maximum number of retries in case of failure

checklist to see if your design is good

[] The application is as decoupled as it can be
[] Nothing can be taken out of an existing container.
[] Every container is transient and is able to react properly when other containers are transient
[] Chaos Monkey can run without my users noticing it
[] Every component can be scaled to meet the workload

Further reading:
K8s CRD: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/
Resource: https://training.linuxfoundation.org/training/kubernetes-for-developers/

DEV Community