EKS cluster upgrade fail with Kubelet version of Fargate pods must be updated to match cluster version!!

You might encounter the following error message within the AWS EKS console when attempting to upgrade the Kubernetes version of your cluster.

Kubelet version of Fargate pods must be updated to match cluster version 1.25 before updating cluster version; Please recycle all offending pod replicas.

For instance, if you are in the process of incrementally upgrading the Kubernetes version from 1.24 to 1.27.

The upgrade from version 1.24 to 1.25 will proceed without encountering errors. However, please note that it won’t automatically update the kubelet version of the Fargate nodes/Pods until a restart is performed.

But when you check Fargate nodes versions,

Or when you check from terminal,

# To list all nodes
kubectl get nodes

Solutions..

If the Fargate pods’ versions are not compatible with the upgraded EKS cluster, you might need to recreate the pods using a compatible version.

• You can do this by updating the deployment or stateful set YAML files with the correct version and applying the changes to the cluster. Kubernetes will automatically create new pods with the updated version.

• You can get all deployment, statefulset from all namespaces and restart it as shown below.

# To list all deployments
kubectl get deployment -A

# To list all deployments
kubectl get statefulset -A

# To restart deployments
kubectl rollout restart deployment coredns -n kube-system
kubectl rollout restart deployment metrics-server -n kube-system
kubectl rollout restart deployment aws-load-balancer-controller -n kube-system
kubectl rollout restart deployment ebs-csi-controller -n kube-system

# To restart statefulset
kubectl rollout restart statefulset adot-collector -n fargate-container-insights

Please note that this solution will work exclusively with Fargate nodes. If you are using other cluster modes, please refer to the links provided at the end of this article for alternative solutions.

Also don’t forget to update kubectl as well.

# To check kubectl version
kubectl version --short --client

Link to update kubectl:
Installing or updating kubectl

For more details about Amazon EKS Kubernetes versions upgrade:
Amazon EKS Kubernetes versions
Updating an Amazon EKS cluster Kubernetes version

I’ll always strive to provide simple explanations. If you found this article useful, follow me for more similar articles in the future. Your support keeps me motivated to create content like this. 🤝😊