Understanding Allocatable Memory and CPU in Kubernetes Nodes

In Kubernetes, allocatable resources refer to the portion of a node’s total resources that can be allocated to running containers or pods. These resources are distinct from the node’s total capacity, which includes all available CPU and memory resources. The key takeaway here is that not all resources on a node are available for use by workloads.

How to find allocatable resources capacity ?

kubectl describe node MY_NODE_1

This command describe your node configuration, the Capacity & Allocatable infomation :

...
Capacity:
  cpu:                12
  ephemeral-storage:  31436544Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             49156248Ki
  pods:               110
Allocatable:
  cpu:                10500m
  ephemeral-storage:  28447630903
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             45908120Ki
  pods:               110
...

As we can see , my node has 12 CPU and only 10.5 CPU is allocatable , 1.5 CPU is reserved for my system + kubelet which is a lot . For the memory, the node has 49Gb and 4Gb are reserved for the OS & kube system , the remain 45Gb is available for the containers.

Understanding Node Capacity and Allocatable Resources

Let’s break down how Kubernetes calculates allocatable resources:

• Total Node Capacity: This is the sum of all the physical CPU and memory resources available on the node. It represents the maximum capacity the node can provide for running containers.
• *Kubernetes system daemons Reserved resources *: Kubernetes sets aside a portion of the node’s resources for system daemons, such as the kubelet, container runtime, and system monitoring tools. These resources are reserved to ensure that critical cluster components can function effectively.
• Node OS Overhead: Some resources are consumed by the operating system itself and are not available for workloads. This includes the kernel, system processes, and filesystem caches. The formula to calculate allocatable resources is:

Allocatable = Total Node Capacity - System Reserved Resources - Kube system daemons Reserved resources

Why Allocatable Resources Matter ?

For several reasons:

• Resource Guarantees: Kubernetes ensures that pods receive the resources they request, and allocatable resources are what Kubernetes uses to make these guarantees. If there are not enough allocatable resources on a node to meet a pod’s requirements, scheduling that pod will fail.
• Efficient Resource Utilization: Efficiently allocating resources is essential for maximizing node utilization and optimizing costs. By managing allocatable resources effectively, you can prevent resource waste and overcommitting nodes.
• Node Stability: If a node runs out of allocatable resources, it can become unstable or unresponsive. Properly managing allocatable resources helps maintain node stability and prevents cluster disruptions.

What quota should you use ?

The following article highlight the quota used by the GCP,AWS & Azure.

Allocatable memory and CPU in Kubernetes Nodes

Pods deployed in your Kubernetes cluster consume resources such as memory, CPU and storage. However, not all resources in a Node can be used to run Pods.

learnk8s.io

How to configure the Node Allocatable resources ?

1- Add the --kube-reserved flag to the kubelet command line with the desired CPU memory reservation value. For example, to reserve 500m CPU memory (0.5 CPU cores), add the following line to the [Service] section of the file:

Environment="KUBELET_EXTRA_ARGS=--kube-reserved=cpu=500m,memory=100Mi"

Make sure to specify the desired value according to your requirements.

2- Save the file and exit the text editor.

3- Reload the systemd service configuration to apply the changes:

sudo systemctl daemon-reload

4- Restart the kubelet service to apply the new configuration:

sudo systemctl restart kubelet

5- Verify that the kubelet has started successfully:

sudo systemctl status kubelet

The kubelet should now be running with the updated CPU memory reservation. Repeat these steps on each node where you want to change the CPU memory reservation.

Keep in mind that modifying kubelet configuration on running production clusters can impact node performance, so be cautious and test changes in a controlled environment first. Additionally, ensure that you have backups and a rollback plan in case the changes cause any issues.