Diagnosing Docker Container Healthcheck Failures: Symptoms, Causes, and Fixes

Recognizable Condition

When you run docker ps you see the container’s STATUS column showing something like Up 5 minutes (unhealthy) and the HEALTH column displays unhealthy. The underlying healthcheck command is returning a non‑zero exit code, timing out, or failing to execute.

Cause/Diagnostic Table

Symptom observed	Likely cause	Quick diagnostic check
Healthcheck logs show “permission denied”	The script/binary referenced in HEALTHCHECK is not executable or lacks proper permissions	Run docker exec -it ls -l /path/to/healthcheck.sh inside the container
Healthcheck exceeds its timeout (default 30s)	The check takes longer than allowed (e.g., a slow curl or a heavy script)	Execute the healthcheck manually with timing: time docker exec
Network‑dependent healthcheck fails (e.g., curl returns “Connection refused”)	The application is not listening on the expected port/interface inside the container	From inside the container, test the endpoint: docker exec -it curl -v http://localhost:/health
Healthcheck command itself fails immediately with “exec format error” or “not found”	The command syntax is wrong, the binary is missing, or the image architecture mismatches	Inspect the HEALTHCHECK instruction: docker inspect --format='{{.Config.Healthcheck}}' and try the command directly

Ordered Checks

1. Confirm unhealthy state: docker inspect --format='{{json .State.Health}}' and note the Status and Log fields.
2. Retrieve the exact healthcheck command: docker inspect --format='{{.Config.Healthcheck.Test}}' (returns a JSON array; the first element is usually "CMD").
3. Run the command inside the container: docker exec -it . Capture the exit code (echo $?) and output.
4. Check file permissions if the command is a script: docker exec -it ls -l /path/to/script.sh. Ensure the owner has execute permission (-x).
5. Measure execution time: time docker exec . Compare to the configured timeout (default 30s,可在 HEALTHCHECK 中通过 --timeout=... 调整).
6. Validate network reachability for HTTP‑based checks: docker exec -it curl -v http://localhost:/health. Look for HTTP 200 or the expected success condition.
7. Review container logs for application errors that might cause the healthcheck to fail: docker logs .

Fixes Tied to Findings

• Permission denied: Rebuild the image with proper permissions, e.g., add RUN chmod +x /usr/local/bin/healthcheck.sh in the Dockerfile, then redeploy.
• Timeout exceeded: Either optimize the healthcheck (reduce work, add caching) or increase the timeout in the HEALTHCHECK instruction: HEALTHCHECK --interval=30s --timeout=10s --retries=3 CMD .... Remember that changing the healthcheck requires a new image build.
• Network unreachable: Ensure the application binds to 0.0.0.0 (or the appropriate interface) and listens on the expected port. Check the Dockerfile or entrypoint for EXPOSE and runtime flags like -p 8080:8080. If the service starts slowly, consider adding a start-period to the healthcheck: HEALTHCHECK --start-period=40s ....
• Malformed command: Correct the syntax in the Dockerfile. A valid form is HEALTHCHECK CMD curl -f http://localhost:8080/health || exit 1. Avoid shell operators unless you wrap the command in sh -c "...".

Escalation Criteria

Escalate to the platform or imaging team when:

• The healthcheck continues to fail after verifying permissions, timeout, and network connectivity.
• The container logs show crashes or OOM kills that are unrelated to the healthcheck but cause the check to never run.
• You need to modify the base image or entrypoint to fix the healthcheck and lack authority to rebuild images.
• Repeated healthcheck failures trigger orchestration actions (e.g., Kubernetes pod restarts) that impact service availability.

In these cases, collect the output of docker inspect , the Dockerfile, and the exact healthcheck command, then open a ticket with the image‑maintenance team.