Diagnosing Docker Container Restart Loops from OOM Kills (Exit 137)

Symptom: docker ps shows a container repeatedly moving between Up and Restarting. docker inspect reports "ExitCode": 137 and "OOMKilled": true. The useful takeaway is that exit code 137 means the kernel’s out-of-memory (OOM) killer sent SIGKILL to the container’s main process because it reached a memory ceiling. The fix is to find whether that ceiling comes from the container’s own limit, host memory pressure, or an application memory leak—then adjust the correct limit instead of only restarting the container.

Recognizable condition

You are likely dealing with an OOM-driven restart loop when all of the following are true:

• docker ps shows Restarting with a short, repeating Up X seconds cycle.
• docker inspect shows State.OOMKilled as true.
• State.ExitCode is 137 (128 plus signal 9, SIGKILL).
• Application logs stop abruptly without a clean shutdown message.

Terms used below: a cgroup is the Linux kernel mechanism that tracks and limits a container’s memory, CPU, and I/O. RSS is resident set size, the portion of a process’s memory held in physical RAM. OOMKilled is Docker’s field recording that the kernel OOM killer terminated the container.

Cause and diagnostic table

Observed condition	Likely cause	First check
Container memory limit is very low, or the host is memory-constrained	Container memory ceiling too small for the workload	docker inspect Memory and MemorySwap fields
Main process spawns workers or sidecars inside the same container	Child processes push combined memory past the parent’s limit	docker stats and cgroup memory usage
Host is under heavy memory pressure with little free RAM	Host OOM pressure triggers kills before Docker limits matter	Host kernel logs (dmesg or journalctl)
Memory use climbs steadily over hours or days	Application memory leak or unbounded allocation	RSS trend in docker stats over time

Ordered checks

1. Confirm the OOM kill on the Docker host

   Run on the Docker host. You need membership in the docker group or sudo.

   docker inspect 

   Check State.OOMKilled and State.ExitCode. If OOMKilled is false but exit code is still 137, another process sent SIGKILL; correlate with kernel logs before assuming OOM. This command is read-only and carries no state-change risk.

2. Read the container’s configured memory ceiling

   docker inspect --format='Memory={{.HostConfig.Memory}} MemorySwap={{.HostConfig.MemorySwap}} Reservation={{.HostConfig.MemoryReservation}}' 

   Values are in bytes. A value of 0 means that setting is unset. If Memory is small relative to the application’s normal use, the container limit is the likely culprit. Read-only.

3. Compare actual usage against cgroup limits

   On cgroup v1 hosts, inspect:

   cat /sys/fs/cgroup/memory/docker//memory.usage_in_bytes
   cat /sys/fs/cgroup/memory/docker//memory.limit_in_bytes

   On cgroup v2 hosts, paths vary by systemd setup, commonly under /sys/fs/cgroup/docker// with memory.current and memory.max. These checks require root or sudo and are read-only. If usage is at or near the limit, the container is constrained.

4. Check host kernel logs for OOM messages

   dmesg | grep -i "out of memory"

   or

   journalctl -k | grep -i "oom-kill"

   Run on the host with sudo. Look for messages naming the container’s process ID or image. This confirms the kernel, not Docker itself, terminated the process. Read-only.

5. Watch live memory behavior

   docker stats 

   Run on the host with docker group access or sudo. Compare MEM USAGE / LIMIT. Note that docker stats may report different numbers than cgroup files; treat it as a trend indicator, not an exact audit. Read-only.

Fixes tied to findings

1. Container memory limit is too low

Raise the limit on a running container:

docker update --memory=512m --memory-swap=1g 

--memory sets the RAM ceiling; --memory-swap sets the total memory-plus-swap ceiling. Set both intentionally. Changing only one can leave an unexpected lower effective cap and produce OOM kills even after the change.

For Compose deployments:

services:
  app:
    image: myapp:1.0
    mem_limit: 512m
    memswap_limit: 1g

Rollback: docker update changes container state immediately. To revert, run docker update again with the previous byte values, or redeploy with the earlier Compose file.

2. Child processes exceed the combined limit

If the container’s entrypoint spawns worker processes, the limit applies to all of them together. Either reduce worker concurrency in the application configuration or raise the container limit as shown above. Do not split workers into separate containers unless you also assign each container its own limit.

3. Host memory pressure

Run free -m on the host. If free RAM is consistently near zero, raising container limits will not help; the host itself is the constraint. Reduce host load, stop unused containers, or add swap space. Recheck kernel logs after the change.

4. Application memory leak

If RSS climbs steadily until the kill, the limit is acting as a circuit breaker, not the root cause. Profile the application or review unbounded caches and queues. A higher limit only delays the restart loop.

Escalation criteria

Escalate to infrastructure or platform engineering when:

• The OOM kill persists after doubling the container limit and confirming host memory is available.
• The kernel OOM killer targets the Docker daemon or host processes, not just containers.
• Cgroup paths or usage values differ between v1 and v2 hosts in a way that prevents reliable diagnosis.
• The workload is production-critical and restarting is not an acceptable interim mitigation.

Limitations and how to verify the result

Exit code 137 strongly suggests an OOM kill, but it is not proof by itself; always confirm with OOMKilled: true and a kernel log message. After applying a fix, verify by running the container under normal load for several minutes and checking:

• docker inspect shows OOMKilled as false.
• docker stats shows memory usage stable and below the configured limit.
• docker ps no longer shows repeated Restarting cycles.

Remember that docker stats can differ from cgroup reports, and cgroup file paths differ between cgroup v1 and v2. If the container still exits with 137 after the limit is raised, either the limit is still too low or the application has a leak that no limit change will resolve.