Diagnosing Docker Container Healthcheck Failures in Production

Recognizable condition

A container that is repeatedly restarted (or stays in unhealthy state) while the application inside appears to be running normally. Docker’s docker ps shows a health status of unhealthy and the RESTARTS column climbs quickly.

Useful takeaway

By isolating the healthcheck execution from the container’s main process, you can pinpoint whether the failure stems from the check itself, resource limits, networking, or the application’s startup timing.

Short cause/diagnostic table

Symptom	Likely cause	Key evidence
Healthcheck fails immediately with “executable file not found”	Incorrect or missing CMD/ENTRYPOINT for the healthcheck command	Exit code 127, log shows exec: "curl": executable file not found
Repeated failures until container restarts, then passes	Healthcheck interval shorter than application startup time	FailingStreak grows, then resets after a restart; logs show connection refused early on
Healthcheck times out after the configured --health-timeout	Healthcheck script exceeds time limit due to heavy work or low CPU/memory limits	Log contains health check timed out; docker stats shows CPU/Memory near limits
Healthcheck returns connection refused or network unreachable	Service bound to localhost but container network isolates it (e.g., --network none or custom bridge)	Log shows curl: (7) Failed to connect to localhost port 8080: Connection refused

Ordered checks

1. Inspect the health status

   Run on the Docker host with sufficient privileges to query the daemon:

   docker inspect --format='{{json .State.Health}}' <container-name-or-id>

   Look for:

   • Status: "unhealthy"
   • FailingStreak: number of consecutive failures
   • Log: contains the last output and exit code.

   Risk: None – read‑only.

2. Re‑run the healthcheck inside the container

   Execute the exact command Docker uses (found in the image’s healthcheck or via docker inspect):

   docker exec -it <container> sh -c '<healthcheck-cmd>'

   Example: if the healthcheck is CMD ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"], run:

   docker exec -it webapp sh -c 'curl -f http://localhost:8080/health || exit 1'

   Observe the exit code and any error output.

   Permission: Requires ability to exec into the container (usually root or a user in the docker group).

   Risk: None – read‑only; may temporarily increase load if the check is heavy.

3. Check timing relative to startup

   Determine the container’s start time and the healthcheck interval:

   docker inspect --format='{{.State.StartedAt}} {{.Config.Healthcheck.Interval}}' <container>

   If the interval (e.g., 5s) is less than the typical application warm‑up time (e.g., 20 s), the probe will fire before the service is ready.

   Risk: None.

4. Verify resource limits during the check

   While the healthcheck is running (you can loop it briefly), monitor usage:

   docker stats <container> --no-stream

   Look for CPU% or MEM% approaching the limits set via --cpus or --memory. High usage combined with a timeout indicates the script is too heavy for the allocation.

   Risk: None.

5. Confirm network reachability of the probed endpoint

   If the healthcheck uses localhost or 127.0.0.1, ensure the service is bound to that address and the container’s network mode permits loopback:

   docker inspect --format='{{.HostConfig.NetworkMode}}' <container>

   For none or a custom bridge without localhost publishing, the check will fail. Test with:

   docker exec -it <container> nc -zv localhost 8080

   Replace nc with telnet or curl -I as needed.

   Risk: None.

Fixes tied to findings

• Incorrect CMD/ENTRYPOINT: Rewrite the Dockerfile to provide a valid healthcheck command, e.g.:

HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD curl -f http://localhost:8080/health || exit 1

Rebuild the image (docker build -t myapp:fixed .) and redeploy the container.

• Interval too short: Increase --interval to exceed the application’s startup time (measure startup with docker logs or an init script). Example:

HEALTHCHECK --interval=45s --timeout=5s --retries=3 \
  CMD curl -f http://localhost:8080/health || exit 1

• Resource‑related timeout: Either lighten the healthcheck script (e.g., use a simple TCP check instead of a heavy SQL query) or raise the container’s CPU/memory limits:

docker run -d --name webapp \
  --cpus="2.0" --memory="512m" \
  --health-cmd="curl -f http://localhost:8080/health" \
  myapp:latest

• Network isolation: Change the network mode to allow loopback, or bind the service to the container’s IP address and adjust the healthcheck accordingly:

# Use the default bridge (allows localhost)
docker run -d --name webapp --network bridge myapp:latest

# Or, if using a custom network, expose the port and check via the container IP:
docker run -d --name webapp --net mynet \
  --health-cmd="curl -f http://$(hostname -i):8080/health" \
  myapp:latest

Escalation criteria

If after performing the ordered checks the healthcheck still fails:

1. Collect the full health log: docker inspect --format='{{json .State.Health.Log}}' <container> and attach it to a ticket.
2. Verify the image’s entrypoint and healthcheck layers with docker history <image> to ensure no unintended overrides.
3. Consider disabling the healthcheck temporarily (--health-cmd=none) to see if the application runs stably; if it does, the fault is definitely in the check logic or its environment.
4. If the application itself is crashing (evident from docker logs showing stack traces or exit codes), treat it as an application bug rather than a healthcheck issue and involve the development team.

Only escalate to infrastructure or platform teams when the evidence points to Docker daemon misconfiguration, host‑level resource exhaustion, or kernel‑level restrictions (e.g., seccomp/AppArmor blocking the healthcheck binary).