RHEL Cgroup Memory Limits and OOM Killer Scoring Interoperability

Answer

During a localized OOM event (when a cgroup’s memory usage exceeds its limit), the kernel first runs the cgroup‑local OOM killer. It selects a victim based on the task’s oom_score (resident set size proportion within that cgroup) plus the task’s oom_score_adj. The global OOM killer is only consulted if the local killer cannot free enough memory (e.g., no killable tasks inside the cgroup) or if the cgroup has no memory limit.

When the system is under global memory pressure but a particular cgroup remains below its limit, the global OOM killer may still target a task inside that cgroup. Its decision uses the system‑wide oom_score (RSS proportion of total memory) plus oom_score_adj. The cgroup limit does not shield tasks from global OOM selection.

Confirmed facts

• Cgroup‑local OOM: victim = oom_score (cgroup‑relative RSS) + oom_score_adj.
• Global OOM: victim = oom_score (system‑wide RSS) + oom_score_adj.
• oom_score_adj has the same additive effect in both contexts (negative = less likely to be killed, positive = more likely).
• The global OOM killer is invoked after a failed local OOM attempt or when no cgroup limit exists.

Verification steps (RHEL 8/9, cgroup v2)

1. Confirm cgroup version: mount | grep cgroup2 or check for /sys/fs/cgroup/unified.
2. Create a test cgroup and set a memory limit, e.g.:

# mkdir -p /sys/fs/cgroup/unified/test
echo 200M > /sys/fs/cgroup/unified/test/memory.max

1. Run a memory‑consuming process inside the cgroup and adjust its oom_score_adj (e.g., -500 to protect, +500 to expose):

# echo $$ > /sys/fs/cgroup/unified/test/cgroup.procs
# stress --vm-bytes 150M --vm-keep &  # adjust size to stay under limit
# echo -500 > /proc/$$/oom_score_adj

1. Exhaust the cgroup’s memory to trigger a local OOM (e.g., increase allocation beyond limit) and observe the killed task via dmesg | grep -i 'killed process' or the cgroup’s OOM log:

# stress --vm-bytes 250M --vm-keep  # exceeds 200M limit
# dmesg | grep -i 'killed process'
# cat /sys/fs/cgroup/unified/test/memory.oom_control

1. Repeat the test while generating global memory pressure outside the cgroup (e.g., another stress instance in the root cgroup) to see whether the global OOM killer selects a task based on system‑wide oom_score despite the cgroup limit.

If the local OOM kills the expected task based on cgroup‑relative RSS and oom_score_adj, the behavior is confirmed. If the global OOM kills a task inside the limited cgroup under system pressure, it confirms that cgroup limits do not shield from global OOM.

Missing diagnostic detail

To refine the recommendation, please confirm whether your system is using the default cgroup v2 hierarchy (/sys/fs/cgroup/unified) or a legacy cgroup v1 setup, as the OOM interaction differs between the two.