Diagnosing Kubernetes Pods Stuck in CrashLoopBackOff

The CrashLoopBackOff Cycle

A CrashLoopBackOff status is not an error itself, but a signal from the kubelet (the node agent) that a container is repeatedly crashing. To prevent the node from wasting resources on a failing process, Kubernetes implements an exponential backoff delay, increasing the time between restart attempts.

The goal of troubleshooting this state is to identify whether the failure is occurring during the initialization phase (configuration/permissions) or the runtime phase (resource exhaustion/health checks).

Rapid Diagnostic Matrix

Use this table to correlate the kubectl describe pod output with the most likely root cause.

Exit Code / Reason	Common Cause	Primary Diagnostic Tool
Exit Code 1 (Error)	Application crash, missing config, or failed DB connection	kubectl logs --previous
Exit Code 137 (OOMKilled)	Container exceeded its memory limit	kubectl describe pod
Exit Code 127 (Not Found)	Incorrect entrypoint path or missing binary in image	kubectl logs
Liveness Probe Failure	App is healthy but probe is too aggressive or timing out	kubectl describe pod (Events)

Step-by-Step Investigation Workflow

Perform these checks in order. Stop once you identify the specific failure point.

1. Inspect the Pod State and Events

Run the following command to see the lifecycle history of the pod. You need permissions to get pod details in the target namespace.

kubectl describe pod <pod-name> -n <namespace>

Look for the Containers section. Specifically, check the Last State. If the reason is OOMKilled, the kernel terminated the process because it exceeded the resources.limits.memory defined in the manifest.

2. Retrieve Logs from the Previous Instance

Because the current container is likely restarting or waiting in backoff, standard logs may be empty. Use the --previous flag to pull the logs from the container that just crashed.

kubectl logs <pod-name> -n <namespace> --previous

What to look for:

• Stack traces indicating missing environment variables.
• Connection timeouts to external dependencies (databases, APIs).
• Permission denied errors when accessing mounted volumes.

3. Analyze Liveness Probe Timing

If the logs show the application started successfully but the pod restarts every few minutes, check the Events section of the describe output for Liveness probe failed.

This often happens when the initialDelaySeconds is shorter than the application's actual boot time. The kubelet kills the container before it has a chance to become ready.

Applying the Fixes

Based on your findings, apply the corresponding correction to your deployment manifest.

Scenario A: Memory Exhaustion (OOMKilled)

If the pod is OOMKilled, avoid simply increasing the limit to a massive number. First, check if the app has a memory leak. If the usage is legitimate, update the resource limits:

# Example snippet in deployment.yaml
resources:
  limits:
    memory: "512Mi" # Increase from 256Mi
  requests:
    memory: "256Mi"

Scenario B: Application Configuration Error

If logs indicate a missing environment variable, ensure the ConfigMap or Secret is correctly referenced. If the variable is missing, the pod will crash immediately upon startup.

Scenario C: Probe Misconfiguration

If the app takes 30 seconds to start but the probe starts at 5 seconds, increase the delay:

# Example snippet in deployment.yaml
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 45 # Allow more time for boot-up

Verification and Rollback

After applying a change via kubectl apply -f manifest.yaml, monitor the pod restart count:

kubectl get pods -w

The RESTARTS column should stop incrementing. If the pod continues to crash or enters a Pending state due to insufficient node resources (after increasing memory limits), roll back the change immediately:

kubectl rollout undo deployment/<deployment-name>

Escalation Criteria

If the following conditions are met, escalate the issue to the image maintainer or infrastructure team:

• The container crashes with Exit Code 127 even though the binary exists in the Dockerfile.
• The pod crashes only on specific nodes (suggesting node-level kernel or storage driver issues).
• Logs are completely empty, and the container exits instantly without any stdout/stderr output.