Diagnosing and Resolving Linux OOM Killer Activations

Recognizable Condition

The kernel log shows a line such as:

Out of memory: Kill process 12345 (java) score 421 or sacrifice child
Killed process 12345 (java) total-vm:8388608kB, anon-rss:4194304kB, file-rss:0kB, shmem-rss:0kB

This indicates the OOM (Out of Memory) killer selected a process for termination because the system could not satisfy a memory allocation request. The message appears in dmesg, /var/log/kern.log, or the systemd journal (journalctl -k).

Cause and Diagnostic Quick-Reference

Symptom	Likely Cause	First Check
OOM message with low MemFree and SwapFree at 0	Physical RAM exhausted; swap full or missing	free -m and swapon --show
OOM despite free memory shown by free	Per-cgroup or per-process limit hit (memory.max, ulimit -v)	cat /sys/fs/cgroup/.../memory.max, ulimit -v
Repeated kills of the same workload	Application memory leak or unbounded cache growth	pidstat -r -p <PID> 1 or pmap -x <PID>
OOM during fork() with plenty of RAM	vm.overcommit_memory=2 with insufficient CommitLimit	cat /proc/meminfo | grep -E 'CommitLimit|Committed_AS'
OOM in container/VM host	Host overcommit vs. guest reservation mismatch	Compare host free with guest configured memory

Ordered Diagnostic Checks

1. Confirm System‑Wide Memory and Swap State

Run as any user (no special permissions needed):

free -m
swapon --show

Expected: Available column should be non‑trivial (hundreds of MB). SwapFree should be > 0 if swap is configured. If swap shows 0B, the system has no swap device or file.

Risk: None; read‑only commands.

2. Examine the Exact OOM Event in Kernel Log

Run as root or with sudo:

journalctl -k -g "Out of memory" --since "1 hour ago"
# or
dmesg -T | grep -i "out of memory"

Note the PID, process name, oom_score, and the total‑vm / anon‑rss values. These tell you whether the killed process was a legitimate memory hog or a victim of bad luck.

3. Check for Cgroup v2 Memory Limits

If the process runs under systemd (most services), inspect its slice:

systemctl status <service-name>
cat /sys/fs/cgroup/system.slice/<service-name>.service/memory.max
cat /sys/fs/cgroup/system.slice/<service-name>.service/memory.current

For user‑session scopes:

cat /sys/fs/cgroup/user.slice/user-<UID>.slice/session-<ID>.scope/memory.max

Finding: If memory.max shows a finite value (not max) and memory.current is near that limit, the OOM was triggered by the cgroup limit, not system‑wide exhaustion.

4. Inspect Overcommit Policy and Commit Accounting

sysctl vm.overcommit_memory
cat /proc/meminfo | grep -E 'CommitLimit|Committed_AS|MemTotal|MemAvailable|SwapTotal|SwapFree'

Interpretation:

• vm.overcommit_memory = 0 (default): heuristic overcommit; kernel may refuse allocations that push Committed_AS far over CommitLimit.
• vm.overcommit_memory = 1: always overcommit; OOM killer runs only when physical pages truly exhausted.
• vm.overcommit_memory = 2: strict overcommit; allocations fail when Committed_AS ≥ CommitLimit (CommitLimit = SwapTotal + RAM × vm.overcommit_ratio/100).

If Committed_AS approaches CommitLimit under mode 2, legitimate allocations fail with ENOMEM and the OOM killer may still fire for page faults.

5. Identify the Offending Process’s Memory Growth

For a recurring PID (or its replacement), track RSS over time:

pidstat -r -p <PID> 5 10
# or, for a one‑time snapshot:
pmap -x <PID> | tail -20

Look for steady RSS climb without plateau. Java heaps, Go runtime, and interpreted languages with unbounded caches are common culprits.

Fixes Tied to Findings

Finding: No Swap Configured

Fix: Add a swap file (immediate, no repartitioning). Run as root:

fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap defaults 0 0' >> /etc/fstab

Verification: swapon --show lists the new file; free -m shows SwapTotal > 0.

Risk: Swap on slow storage (HDD, network block) can cause latency spikes. Prefer NVMe/SSD. Size: 1–2× RAM for hibernation; 1–4 GB often sufficient for OOM buffer on servers.

Finding: Cgroup Memory Limit Too Low

Fix: Raise or remove the limit for the service. For a systemd service:

systemctl set-property <service-name>.service MemoryMax=8G
# or remove limit:
systemctl set-property <service-name>.service MemoryMax=infinity

Verification: cat /sys/fs/cgroup/.../memory.max shows new value; restart service to apply.

Risk: Removing limits lets one service starve others. Prefer setting a generous ceiling (e.g., 80% of RAM) rather than infinity.

Finding: Application Memory Leak

Fix: This requires application‑level work. Immediate mitigations:

• Restart the service periodically via systemd: RuntimeMaxSec=12h in [Service] section.
• Set a cgroup memory.high (soft limit) to trigger reclaim before hard OOM: systemctl set-property service-name.service MemoryHigh=6G.
• Enable heap profiling (Java: -XX:+HeapDumpOnOutOfMemoryError; Go: pprof; Python: tracemalloc).

Verification: After deploy, monitor memory.current for 24 h; it should stabilize.

Finding: Strict Overcommit (Mode 2) Starving Fork‑Heavy Workloads

Fix: Switch to heuristic overcommit or increase overcommit_ratio. As root:

sysctl -w vm.overcommit_memory=0
# or keep mode 2 but raise ratio:
sysctl -w vm.overcommit_ratio=80
# persist:
echo 'vm.overcommit_memory=0' > /etc/sysctl.d/99-overcommit.conf
sysctl --system

Verification: cat /proc/meminfo shows CommitLimit increased; fork‑heavy workloads (e.g., Apache prefork, CI runners) stop seeing ENOMEM.

Risk: Mode 0/1 allows overcommit; a runaway process can still trigger OOM later. Test in staging under load.

Escalation Criteria

Escalate to deeper investigation or hardware change when:

1. OOM recurs after applying the above fixes — enable a core dump for the killed process to inspect heap state:

   echo '/var/crash/core.%e.%p' > /proc/sys/kernel/core_pattern
   mkdir -p /var/crash
   chmod 1777 /var/crash
   ulimit -c unlimited  # in service environment or /etc/security/limits.conf

   Analyze with gdb /path/to/binary /var/crash/core.<PID> or language‑specific tools.
2. Multiple unrelated processes killed in quick succession — indicates systemic memory pressure; review total RAM vs. workload demand. Consider adding physical memory or moving workloads to another node.
3. OOM with MemAvailable > 20% of RAM — suggests fragmentation or zone imbalance. Check cat /proc/buddyinfo and cat /proc/pagetypeinfo. May require kernel upgrade or vm.compaction_proactiveness tuning.
4. Container host OOM killing guest workloads — host overcommit accounting doesn’t see guest ballooning. Use memory limits on containers (docker run --memory=4g) and disable host swap for dedicated container nodes.

Limitations and Practical Verification

• The OOM killer’s oom_score is heuristic; a critical system process (sshd, systemd-journald) can still be chosen. Protect key services with echo -1000 > /proc/<PID>/oom_score_adj (requires root, persists only until restart). For systemd services, use OOMScoreAdjust=-1000 in the unit file.
• vm.overcommit_memory changes affect the entire system; test in a staging environment that mirrors production memory pressure patterns.
• Swap files on btrfs or zfs require specific options (no COW, no compression). Verify with lsattr /swapfile showing C (no‑copy‑on‑write) on btrfs.
• Memory cgroup limits apply to the entire cgroup tree; a limit on a parent slice affects all children. Use systemd-cgtop to visualize hierarchy usage.

Verification Checklist After Remediation

1. Reproduce the peak load (traffic replay, batch job, test suite) for at least 30 minutes.
2. Confirm no new OOM messages: journalctl -k -g "Out of memory" --since "30 minutes ago" returns nothing.
3. Verify the target process RSS stabilizes: pidstat -r -p <PID> 10 6 shows flat or sawtooth pattern, not monotonic climb.
4. Ensure swap usage remains low (< 10% of swap) during normal operation; high steady swap indicates undersized RAM.
5. Check that latency SLOs are met; excessive swapping or compaction can degrade tail latency even without OOM.