CreateContainerConfigError and CreateContainerError are two of the most prevalent Kubernetes errors found in cloud-native applications.
CreateContainerConfigError is an error happening when the configuration specified for a container in a Pod is not correct or is missing a vital part.
CreateContainerError is a problem happening at a later stage in the container creation flow. Kubernetes displays this error when it attempts to create the container in the Pod.
In this article, you will learn:
- What is Kubernetes CreateContainerConfigError?
- What is Kubernetes CreateContainerError?
- Kubernetes container creation flow
- Common causes for CreateContainerError and CreateConfigError
- How to troubleshoot both errors
- How to detect both errors in Prometheus
What is CreateContainerConfigError?
During the process to start a new container, Kubernetes first tries to generate the configuration for it. In fact, this is handled internally by calling a method called generateContainerConfig, which will try to retrieve:
- Container command and arguments
- Relevant persistent volumes for the container
- Relevant ConfigMaps for the container
- Relevant secrets for the container
Any problem in the elements above will result in a CreateContainerConfigError.
What is CreateContainerError?
Kubernetes throws a CreateContainerError when there’s a problem in the creation of the container, but unrelated with configuration, like a referenced volume not being accessible or a container name already being used.
Similar to other problems like CrashLoopBackOff, this article only covers the most common causes, but there are many others depending on your current application.
How you can detect CreateContainerConfigError and CreateContainerError
You can detect both errors by running kubectl get pods:
NAME READY STATUS RESTARTS AGE
mypod 0/1 CreateContainerConfigError 0 11m
As you can see from this output:
- Pod is not ready: container has an error.
- There are no restarts: these two errors are not like CrashLoopBackOff, where automatic retrials are in place.
Kubernetes container creation flow
In order to understand CreateContainerError and CreateContainerConfligError, we need to first know the exact flow for container creation.
Kubernetes follows the next steps every time a new container needs to be started:
- Pull the image.
- Generate container configuration.
- Precreate container.
- Create container.
- Pre-start container.
- Start container.
As you can see, steps 2 and 4 are where a CreateContainerConfig and CreateContainerErorr might appear, respectively.
Common causes for CreateContainerError and CreateContainerConfigError
Not found ConfigMap
Kubernetes ConfigMaps are a key element to store non-confidential information to be used by Pods as key-value pairs.
When adding a ConfigMap reference in a Pod, you are effectively indicating that it should retrieve specific data from it. But, if a Pod references a non-existent ConfigMap, Kubernetes will return a CreateContainerConfigError.
Not found Secret
Secrets are a more secure manner to store sensitive information in Kubernetes. Remember, though, this is just raw data encoded in base64, so it’s not really encrypted, just obfuscated.
In case a Pod contains a reference to a non-existent secret, Kubelet will throw a CreateContainerConfigError, indicating that necessary data couldn’t be retrieved in order to form container config.
Container name already in use
While an unusual situation, in some cases a conflict might occur because a particular container name is already being used. Since every docker container should have a unique name, you will need to either delete the original or rename the new one being created.
How to troubleshoot CreateContainerError and CreateContainerConfigError
While the causes for an error in container creation might vary, you can always rely on the following methods to troubleshoot the problem that’s preventing the container from starting.
Describe Pods
With kubectl describe pod
, you can retrieve the detailed information for the affected Pod and its containers:
Containers:
mycontainer:
Container ID:
Image: nginx
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: CreateContainerConfigError
Ready: False
Restart Count: 0
Limits:
cpu: 3
---
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: myconfigmap
Optional: false
Get logs from containers
Use kubectl logs
to retrieve the log information from containers in the Pod. Note that for Pods with multiple containers, you need to use the –all-containers
parameter:
Error from server (BadRequest): container "mycontainer" in pod "mypod" is waiting to start: CreateContainerConfigError
Check the events
You can also run kubectl get events
to retrieve all the recent events happening in your Pods. Remember that the describe pods command also displays the particular events at the end.
How to detect CreateContainerConfigError and CreateContainerError in Prometheus
When using Prometheus + kube-state-metrics, you can quickly retrieve Pods that have containers with errors at creation or config steps:
kube_pod_container_status_waiting_reason{reason="CreateContainerConfigError"} > 0
kube_pod_container_status_waiting_reason{reason="CreateContainerError"} > 0
Other similar errors
Pending
Pending is a Pod status that appears when the Pod couldn’t even be started. Note that this happens at schedule time, so Kube-scheduler couldn’t find a node because of not enough resources or not proper taints/tolerations config.
ContainerCreating
ContainerCreating is another waiting status reason that can happen when the container could not be started because of a problem in the execution (e.g: No command specified
)
Error from server (BadRequest): container "mycontainer" in pod "mypod" is waiting to start: ContainerCreating
RunContainerError
This might be a similar situation to CreateContainerError, but note that this happens during the run step and not the container creation step.
A RunContainerError most likely points to problems happening at runtime, like attempts to write on a read-only volume.
CrashLoopBackOff
Remember that CrashLoopBackOff is not technically an error, but the waiting time grace period that is added between retrials.
Unlike CrashLoopBackOff events, CreateContainerError and CreateContainerConfigError won’t be retried automatically.
Conclusion
In this article, you have seen how both CreateContainerConfigError and CreateContainerError are important messages in the Kubernetes container creation process. Being able to detect them and understand at which stage they are happening is crucial for the day-to-day debugging of cloud-native services.
Also, it’s important to know the internal behavior of the Kubernetes container creation flow and what is errors might appear at each step.
Finally, CreateContainerConfigError and CreateContainerError might be mistaken with other different Kubernetes errors, but these two happen at container creation stage and they are not automatically retried.
Troubleshoot CreateContainerError with Sysdig Monitor
With Sysdig Monitor’s Advisor, you can easily detect which containers are having CreateContainerConfigError or CreateContainerError problems in your Kubernetes cluster.
Advisor accelerates mean time to resolution (MTTR) with live logs, performance data, and suggested remediation steps. It’s the easy button for Kubernetes troubleshooting!
Try it free for 30 days!
Top comments (0)