When you type kubectl delete pod, the pod is deleted, and the endpoint controller removes its IP address and port (endpoint) from the Services and etcd.
You can observe this with kubectl describe service.
But that's not enough!
Several components sync a local list of endpoints:
- kube-proxy keeps a local list of endpoints to write iptables rules.
- CoreDNS uses the endpoint to reconfigure the DNS entries.
And the same is true for the Ingress controller, Istio, etc.
All those components will (eventually) remove the previous endpoint so that no traffic can ever reach it again.
At the same time, the kubelet is also notified of the change and deletes the pod.
What happens when the kubelet deletes the pod before the rest of the components?
Unfortunately, you will experience downtime because components such as kube-proxy, CoreDNS, the ingress controller, etc., still use that IP address to route traffic.
So what can you do?
Wait!
If you wait long enough before deleting the pod, the in-flight traffic can still resolve, and the new traffic can be assigned to other pods.
How are you supposed to wait?
When the kubelet deletes a pod, it goes through the following steps:
- Triggers the
preStophook (if any). - Sends the SIGTERM.
- Sends the SIGKILL signal (after 30 seconds).
You can use the preStop hook to insert an artificial delay.
You can listen to the SIGTERM signal in your app and wait.
Also, you can gracefully stop the process and exit when you are done waiting.
Kubernetes gives you 30s to do so (configurable).
Should you wait 10 seconds, 20 or 30s?
There's no single answer.
While propagating endpoints could only take a few seconds, Kubernetes doesn't guarantee any timing nor that all of the components will complete it at the same time.
If you want to explore more, here are a few links:
- https://learnk8s.io/graceful-shutdown
- https://freecontent.manning.com/handling-client-requests-properly-with-kubernetes/
- https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
- https://medium.com/tailwinds-navigator/kubernetes-tip-how-to-gracefully-handle-pod-deletion-b28d23644ccc
- https://medium.com/flant-com/kubernetes-graceful-shutdown-nginx-php-fpm-d5ab266963c2
- https://www.openshift.com/blog/kubernetes-pods-life
And finally, if you've enjoyed this thread, you might also like:
- The Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training
- This collection of past threads https://twitter.com/danielepolencic/status/1298543151901155330
- The Kubernetes newsletter I publish every week, "Learn Kubernetes weekly" https://learnk8s.io/learn-kubernetes-weekly
Top comments (0)