Diagnosing Kubernetes CrashLoopBackOff: Exit Codes, Probes & Node Pressure

Problem: Container Restarts Without Stopping

When a pod repeatedly restarts and settles into CrashLoopBackOff, the cluster marks it unhealthy but does not surface the root cause immediately. The kubelet starts the container, it exits, the kubelet waits using exponential backoff, and the cycle repeats. CrashLoopBackOff is not an error itself; it is the backoff state between restarts of a container that keeps dying. The real signal is the previous container's exit code and logs. This guide walks through recognizing the condition, interpreting exit codes, distinguishing probe and scheduling failures, and applying fixes grounded in the pod's actual state.

Diagnostic Table: Exit Codes at a Glance

Exit Code	Likely Cause	Action
1	Application-level error (uncaught exception, config failure)	Review application logs; fix code or configuration
137 (128 + 9)	SIGKILL — usually OOMKilled by kubelet or external kill	Check OOMKilled flag in pod status; adjust resources.limits.memory or fix leaks
126	Container command found but not executable	Verify entrypoint or command syntax; check file permissions
127	Command not found in PATH	Use fully qualified path or adjust PATH in the spec
0 with restartPolicy: Always	Process exits successfully but should run indefinitely (e.g. one-shot job misconfigured)	Review restartPolicy and intended workload type

Ordered Check Sequence

1. Check pod status and restart count: Run kubectl get pods in the affected namespace. Look for a pod whose STATUS is CrashLoopBackOff and note the RESTARTS count.

2. Inspect events and last state: Run kubectl describe pod POD_NAME. Examine the Events section for warnings about image pulls, probe failures, or scheduling. Then find the lastState field — this is where the container's final exit code and OOMKilled flag appear.

3. Retrieve the dying container's output: Run kubectl logs POD_NAME --previous. The --previous flag is essential; without it, kubectl logs shows the current (possibly empty or mid-startup) container instance, which offers little diagnostic value. Expected check: match the exit code reported in lastState with output lines near the end of the log stream.

4. Verify liveness and readiness probes: Review the pod spec's livenessProbe and readinessProbe. A misconfigured probe path, wrong port, or initialDelaySeconds that is too short can cause a liveness probe to fail repeatedly, which restarts the container. Note that readiness probe failures do not trigger restarts — only liveness failures do.

5. Check node-level pressure: If multiple unrelated pods on the same node are crashing together, run kubectl describe node NODE_NAME and look for MemoryPressure, DiskPressure, or PIDPressure conditions. Also check kubelet and container runtime status on that node.

Fixes Mapped to Findings

• Memory exhaustion (exit code 137, OOMKilled=true): Raise resources.limits.memory on the container, but first verify that the node has allocatable memory to spare. A limit increase without a node capacity check can move the problem from pod-level eviction to node-level eviction. Alternatively, profile the application for memory leaks and fix the code.
• Command or permission errors (exit codes 126/127): Correct the entrypoint command or file permissions in the pod spec. Ensure the executable exists at the declared path and is executable.
• Failing liveness probe: Adjust initialDelaySeconds to give the application enough startup time, verify the probe path and port match the running service, and consider a timeoutSeconds value that prevents hanging.
• Image pull failure (often confused with CrashLoopBackOff): If kubectl describe pod shows pull errors and BackOff events for image retrieval, the issue is ImagePullBackOff, not CrashLoopBackOff. Common causes are bad tags, missing registry credentials (configure imagePullSecrets), or nodes that cannot reach a private registry endpoint.

When to Escalate

Move beyond pod-level debugging if any of the following apply:

• Multiple unrelated pods on the same node are crashing simultaneously — suspect node resource pressure, kubelet malfunction, or container runtime issues.
• Crashes correlate with a recent rollout or configuration change. Roll back the change first (e.g. kubectl rollout undo deployment DEPLOYMENT_NAME) and observe whether restarts stabilize.
• The cluster control plane reports a node as NotReady. This indicates a networking or kubelet communication problem that requires platform-team involvement.

Limitations and Verification

Exit code reporting, event wording, and probe defaults vary slightly across Kubernetes versions. Confirm behavior against the version running in your cluster (kubectl version). After applying a fix, watch the pod with kubectl get pods -w for several minutes to confirm that the restart count stops increasing; a single Running state does not guarantee resolution if the process exits shortly after. You can also reproduce the condition safely in a test namespace by deploying a container that exits immediately and confirming that kubectl describe pod shows CrashLoopBackOff with the expected exit code.