Diagnosing Docker Exit Code 137: Tracing OOM Kills from Container to Host

A container that starts cleanly and then dies with exit code 137 within seconds or minutes is almost always a memory problem, not an application bug. Exit code 137 means the process received SIGKILL (128 + 9), and in Docker the most common sender is the kernel's out-of-memory (OOM) killer enforcing a cgroup memory limit. The useful takeaway: before touching application code, confirm which limit was hit — the container's own limit or the host's — because the fixes are completely different.

What the failure looks like

The recognizable pattern is a container that exits quickly after a burst of memory allocation. Application logs often end abruptly, sometimes with an allocation error just before the cutoff. docker events shows kill and die in rapid succession. If a restart policy is set, docker ps -a shows a climbing restart count with exit code 137 each time.

Mapping symptoms to causes

Observation	Likely cause
MEM USAGE sits near LIMIT in docker stats before exit	Container memory limit too low for the workload
RSS climbs steadily over hours with no plateau	Memory leak in the application
dmesg shows OOM kills across multiple unrelated containers	Host memory overcommit; host-level OOM
Kills occur despite free swap on the host	Swap disabled or memory+swap limit reached

Ordered checks

Run these on the Docker host as a user with access to the Docker socket (typically root or the docker group). Kernel log inspection requires host-level read access to dmesg.

1. Confirm the exit code and restart pattern. docker ps -a — look for Exited (137) and note how fast restarts occur.
2. Read the configured limits. docker inspect <container> and check HostConfig.Memory, HostConfig.MemorySwap, and (on older setups) HostConfig.KernelMemory. Also check State.OOMKilled — a value of true confirms the kernel OOM killer terminated the container.
3. Compare usage against the limit. docker stats --no-stream shows MEM USAGE / LIMIT. If usage hugs the limit before each death, the limit itself is the constraint.
4. Check the kernel log. dmesg -T | grep -i oom — the OOM killer log names the cgroup and the process it chose to kill, which tells you whether the kill was scoped to the container's cgroup or a host-wide OOM.
5. Review application metrics. Look for allocation spikes correlated with the exit times — batch jobs, cache warm-up, or large query results are common triggers.

Fixes tied to findings

Limit set too low

Raise the limit and redeploy. In Compose (v2/v3 syntax on Docker Engine with cgroups):

services:
  worker:
    image: example/worker:1.4
    mem_limit: 1g
    memswap_limit: 1g   # equal to mem_limit disables swap for this container

For docker run, the equivalents are --memory=1g --memory-swap=1g. Verify after redeploy with docker inspect that HostConfig.Memory matches the intended value.

Confirmed memory leak

Raising the limit only delays the kill and increases blast radius on shared hosts. Restart the service to recover, reduce batch size or concurrency as a stopgap, and add memory profiling (heap dumps, RSS tracking) to find the leak. Keep the limit in place as a safety net — that is what it is for.

Host overcommit

If dmesg shows the OOM killer choosing victims across several containers, the host itself is out of memory. Free capacity by moving workloads to other nodes, stopping nonessential containers, or adding host memory. On orchestrators, check that the sum of container limits does not grossly exceed node capacity.

Unexpected swap behavior

Setting --memory-swap equal to --memory disables swap for the container. Setting --memory-swap=-1 allows unlimited swap, which can cause heavy disk thrashing and looks like a hang rather than a kill. Pick an explicit value and document why.

Verifying the fix

• Run docker stats under representative load and confirm MEM USAGE stays comfortably below LIMIT.
• Watch docker ps -a: restart count should stop growing and exit codes should be 0.
• Re-check dmesg -T | grep -i oom after the change window — no new entries for the container's cgroup.
• Confirm health checks pass, since rolling redeploys can briefly raise peak memory while old and new containers coexist.

When to escalate

Escalate to the platform or SRE team when any of these hold: repeated 137 kills across multiple replicas or nodes within minutes; evidence of incomplete writes or data corruption; OOM kills hitting system-critical containers such as databases or control-plane components; or memory usage remaining high even after a code fix and a limit increase. Do not tune kernel OOM settings (such as oom_score_adj or vm sysctls) on a shared host without platform approval — that shifts the kill target rather than fixing the cause.

Limitations

Commands and field names above assume a current Docker Engine on Linux with cgroup v1 or v2; exact docker inspect field paths and Compose keys vary slightly by version, so verify against your installed release. State.OOMKilled only reflects the most recent exit, so check it before any manual restart. On systems where dmesg is restricted (kernel.dmesg_restrict=1), you will need elevated privileges or the platform team's logs.