DEV Community

Dean
Dean

Posted on • Originally published at veducate.co.uk on

Deleting AWS EKS Cluster fails – Cannot evict pod as it would violate the pod’s disruption budget

The Issue

I had to remove a demo EKS Cluster where I had screwed up an install of a Service Mesh. Unfortunately, it was left in a rather terrible state to clean up, hence the need to just delete it.

When I tried the usual eksctl delete command, including with the force argument, I was hitting errors such as:

2021-12-21 23:52:22 [!] pod eviction error ("error evicting pod: istio-system/istiod-76f699dc48-tgc6m: Cannot evict pod as it would violate the pod's disruption budget.") on node ip-192-168-27-182.us-east-2.compute.internal
Enter fullscreen mode Exit fullscreen mode

With a final error output of:

Error: Unauthorized
Enter fullscreen mode Exit fullscreen mode

eksctl delete cluster - Cannot evict pod as it would violate the pod's disruption budget - Error Unauthorized

The Cause

Well, the error message does call out the cause, moving the existing pods to other nodes is failing due to the configured settings. Essentially EKS will try and drain all the nodes and shut everything down nicely when it deletes the cluster. It doesn’t just shut everything down and wipe it. This is because inside of Kubernetes there are several finalizers that will call out actions to interact with AWS components (thanks to the integrations) and nicely clean things up (in theory).

To get around this, I first tried the following command, thinking if delete the nodegroup without waiting for a drain, this would bypass the issue:

 eksctl delete nodegroup standard --cluster veducate-eks --drain=false --disable-eviction
Enter fullscreen mode Exit fullscreen mode

This didn’t allow me to delete the cluster however, I still got the same error messages.

The Fix

So back to the error message, and then I realised it was staring me in the face!

Cannot evict pod as it would violate the pod's disruption budget
Enter fullscreen mode Exit fullscreen mode

What is a Pod Disruption Budget? It’s essentially a way to ensure availability of your pods from someone killing them accidentality.

A PDB limits the number of Pods of a replicated application that are down simultaneously from voluntary disruptions. For example, a quorum-based application would like to ensure that the number of replicas running is never brought below the number needed for a quorum. A web front end might want to ensure that the number of replicas serving load never falls below a certain percentage of the total.
Enter fullscreen mode Exit fullscreen mode

To find all configured Pod Disruption Budgets:

kubectl get poddisruptionbudget -A
Enter fullscreen mode Exit fullscreen mode

Then delete as necessary:

kubectl delete poddisruptionbudget {name} -n {namespace}
Enter fullscreen mode Exit fullscreen mode

eks - kubectl get poddisruptionbudgets -A - kubectl delete poddisruptionbudgets

Finally, you should be able to delete your cluster.

eksctl delete cluster - successful

Regards

Follow @Saintdle

Dean Lewis

The post Deleting AWS EKS Cluster fails – Cannot evict pod as it would violate the pod’s disruption budget appeared first on vEducate.co.uk.

Discussion (0)