By Levent Ogut
Kubernetes brought an excellent deployment platform to work on. Even monolithic applications can be run in a container. For some of these monolithic applications and for some microservices, a slow start is a problem. Even if we configure readiness and liveness probes using initialDelaySeconds, this is not an ideal way to do this. For this specific problem, startup probes are developed.
Probes
Probes are executed by kubelet to determine pods' health.
All three types of probes have common settings to configure.
- initialDelaySeconds: How many seconds to wait after the container has started (default: 0)
- periodSeconds: Wait time between probe executions (default: 10)
- timeoutSeconds: Timeout of the probe (default: 1)
- successThreshold: Threshold needed to mark the container healthy. (default: 1)
- failureThreshold: Threshold needed to mark the container unhealthy. (default: 3)
Configuring these parameters vital and crucial.
Exec Probe
Exec probe runs a command inside the container as a health check; the command's exit code determines the success.
startupProbe:
initialDelaySeconds: 1
periodSeconds: 5
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 1
exec:
command:
- cat
- /etc/nginx/nginx.conf
TCP Probe
TCP probe checks if the specified port is open or not; an open port points to success.
startupProbe:
initialDelaySeconds: 1
periodSeconds: 5
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 1
tcpSocket:
host:
port: 80
HTTP Probe
HTTP probe sends an HTTP GET request with defined parameters.
HTTP Probe has additional options to configure.
- host: Host/IP to connect to (default: pod IP)
- scheme: Scheme to use when making the request (default: HTTP)
- path: Path
- httpHeaders: An array of headers defined as header/value tuples.
- port: Port to connect to
Tip: Host header should be set in httpHeaders.
startupProbe:
initialDelaySeconds: 1
periodSeconds: 2
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 1
httpGet:
host:
scheme: HTTP
path: /
httpHeaders:
- name: Host
value: myapplication1.com
port: 80
initialDelaySeconds: 5
periodSeconds: 5
Startup Probes in Kubernetes
Startup probes are a newly developed feature supported as a beta in Kubernetes v.1.18.
These probes are very useful on slow-start applications; it is much better than increasing initialDelaySeconds on readiness or liveness probes.
Startup probe allows our application to become ready, joined with readiness and liveness probes, it can dramatically increase our applications' availability.
Example: Sample Nginx Deployment
Let's deploy Nginx as a sample app and see startup probes in action.
apiVersion: apps/v1
kind: Deployment
metadata:
name: k8s-probes
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
startupProbe:
initialDelaySeconds: 1
periodSeconds: 2
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 1
exec:
command:
- cat
- /etc/nginx/nginx.conf
Create a file and paste the above, if you named the file k8s-probes-deployment.yaml and apply it with kubectl apply -f k8s-probes-deployment.yaml command.
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx
name: nginx
namespace: default
spec:
ports:
- name: nginx-http-port
port: 80
selector:
app: nginx
sessionAffinity: None
type: NodePort
Create a file and paste the above, if you named the file k8s-probes-svc.yaml and apply it with kubectl apply -f k8s-probes-svc.yaml command.
Common Pitfalls for Startup Probes
Although it is great to have such a probe, especially on a legacy application or any application that might take a while to become ready, it is quite important that parameters must be correctly configured. Otherwise, it can break our application and availability.
Restart Loop
Startup probes - if misconfigured- can cause a loop of restarts. Let's assume we have an application running on java, and this application takes a while to become ready. If we don't allow enough time for the startup probe to get a successful response, the kubelet might restart the container prematurely, causing a loop of restarts.
Troubleshooting Startup Probes
After applying the deployment file, we should see the pod is up and running; let's have a look.
kubectl get pods
We can see that the pod is up and running.
NAME READY STATUS RESTARTS AGE
k8s-probes-6cbf7ccbf8-97hz5 1/1 Running 0 7s
Have a look at the events using the kubectl describe pods command.
kubectl describe pods k8s-probes-6cbf7ccbf8-97hz5
Let's scroll down to events.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3s default-scheduler Successfully assigned default/k8s-probes-6cbf7ccbf8-97hz5 to k8s-probes
Normal Pulling 2s kubelet Pulling image "nginx"
Normal Pulled 1s kubelet Successfully pulled image "nginx" in 925.688112ms
Normal Created 1s kubelet Created container nginx
Normal Started 1s kubelet Started container nginx
As you can see, the probe was successful, and no error or warning event is recorded.
Let's check the applied configuration.
kubectl describe pods k8s-probes-6cbf7ccbf8-qcpt7 | grep Startup
We can see that the startup probe is configured with the parameters we have set.
Startup: exec [cat /etc/nginx/nginx.conf] delay=1s timeout=1s period=2s #success=1 #failure=1
Now change the command parameter to /etc/nginx/nginx.conf-dont-exists on the deployment file and apply with kubectl apply -f k8s-probes-deployment.yaml.
Let's check the events of the pod.
kubectl describe pods k8s-probes-5fcc896b6f-97wpg
Let's scroll down to events.
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m default-scheduler Successfully assigned default/k8s-probes-5fcc896b6f-97wpg to k8s-probes
Normal Pulled 13m kubelet Successfully pulled image "nginx" in 944.990287ms
Normal Pulled 12m kubelet Successfully pulled image "nginx" in 972.83673ms
Normal Pulled 12m kubelet Successfully pulled image "nginx" in 958.559546ms
Normal Pulled 11m kubelet Successfully pulled image "nginx" in 1.056812046s
Normal Created 11m (x4 over 13m) kubelet Created container nginx
Normal Started 11m (x4 over 13m) kubelet Started container nginx
Warning Unhealthy 11m (x4 over 13m) kubelet Startup probe failed: cat: /etc/nginx/nginx.conf-dont-exists: No such file or directory
Normal Pulling 9m44s (x5 over 13m) kubelet Pulling image "nginx"
Normal Killing 8m21s (x5 over 12m) kubelet Container nginx failed startup probe, will be restarted
Warning BackOff 3m41s (x13 over 7m16s) kubelet Back-off restarting the failed container
As you can see startup probe executed the command; however, due to a non-existent file, the command returned a non-zero exit code.
Conclusion
Startup probes are very helpful to determine our application has started correctly.
We have explored the config options on a sample Nginx application; we have checked the configuration file as a probe; this is an example of a dynamic configuration generation application, i.e., assume that Nginx configuration is generated dynamically using etcd or similar key-value store.
Although very useful, we have also explored the side effects of the startup probes. Make sure you allow enough time to let your application startup.
Further Reading
- Kubernetes Liveness Probes - Examples & Common Pitfalls
- Kubernetes Readiness Probes - Examples & Common Pitfalls
- Kubernetes Core Probe Documentation
- Configure Liveness, Readiness and Startup Probes
- Kubernetes Container probes Documentation
- Container Lifecycle Hooks Documentation
Photo by Braden Collum on Unsplash
Top comments (0)