Declutter your Kubernetes Cluster

This article will cover the basics to clean up your Kubernetes Cluster.

Prerequisites ✅

• Basic understanding of Kubernetes
• Understanding of Docker: Here

🤔 Understanding the Problem

A Kubernetes Cluster is the home for your applications. This platform is responsible to serve your users with little to no downtime. It only makes sense to keep it clean, yet these are three major issues, that I see:

• Easy to manage: I have been an SRE, and I assure you a decluttered cluster is easier to manage. In the time of fire, each second that you can save counts.
• Reduce resource utilization: Unused deployments take up resources. The resources that could be helpful to other deployments. The scheduler has to work harder to find resources for pods.
• Reduce Cost: Save money. FinOps is a big task, and these unwanted resources mainly add to it!

Ways to Clean up your Cluster 💡

Let's get started!

Before we take a deep dive, you have to keep in mind these points:

• Never perform any of these actions without consulting the team.
• Resource optimization should not hamper functionality.
• Cache saves time.
• Have a backup strategy.

Now, let's talk about the ways to clean up your Cluster.
Like most of the tasks in Kubernetes, there are three ways to achieve this.

• Manual Kubectl commands
• In-built service
• Third-party services

I. Manual Kubectl commands

In this, I will explain different strategies that can be used to optimize and clean your cluster. The rest of the methods are abstractions on top of these.

i. Remove deadweight:
Delete pods that are in Evicted / Error / Completed state. Be cautious to check for stateful pods before deleting them.
You can use this kubectl command to find such resource:

kubectl get pods --all-namespaces -o wide | grep Evicted

ii. Use of in-built autoscaling:
HPA or Horizontal Pod Autoscaling ensures resources are allocated efficiently and reduces the need for manual intervention.
VPA or Vertical Pod Autoscaling helps to avoid overprovisioning and makes the cluster cleaner by optimizing resource allocations.

iv. Tidy ConfigMaps and Secrets:
Update or delete ConfigMaps that are not in use. Doesn't save a lot of space but certainly makes the life of DevOps folks easier.
As per secrets are concerned, the industry standard is to save Kubernetes Secrets in Secret vaults like Gitlab Secrets, AWS KMS, or Hashicorp Vault.

v. Use Daemonsets and PDBs wisely:
Daemonsets can create pods on each node. And PDB can create pods that are tough to kill. Use these wisely!

vi. Labels and Annotations:
Label your resources, it becomes easier for DevOps folks to manage the cluster. Annotations help DevOps to get a better understanding of the ownership of the pod and contact the right person for debugging.

I can add more steps but it would be more about organizing your clustering than decluttering it. Moving on to the next step!

II. Kubernetes Garbage Collector

Kubernetes has an in-built Garbage Collector, read Here. There are multiple options available in the Garbage collector. There are several flags in the Kubernetes configuration file you can set. Digging deep into this here might be the scope of another article.

I will add the references for this at the end.

III. Third-party apps

It is advised against using third-party apps with permission to delete or clean resources. But just to state some services I found for this task:

• Cilium
• KubeDog
• Some less popular tools like Kleaner and more are there.

I have never used any of these services at any of my clusters. I strongly advise against using them without doing a POC around them.

Conclusion

Try to use Kubernetes Garbage Collector over doing manual Kubectl commands. GC is controlled by the Kubelet does the drift is registered by etcd. Running multiple kubectl commands to delete a bunch of resources is never a good idea. Using third-party apps is the last option. Ideally, no service should have admin access on your cluster. There is another option to build a Kubernetes Operator from scratch.

References

• Great commands for the use case: Here
• Source code of K8s/garbagecollector: Here
• Good read to understand GC: Here

If you liked this content you can follow me on Twitter at kitarp29 for more!

Thanks for reading my article :)