Question
Which oom_score_adj values prevent critical service termination in cgroup v2?
Lax Cedar
0 reputation · 29 Aug 2023, 05:00 UTC
135.8K views
Loading…
0 reputation · 29 Aug 2023, 05:00 UTC
In Red Hat Enterprise Linux (RHEL), the Out of Memory (OOM) Killer selects processes for termination based on a calculated score. To protect critical system services from being targeted during memory pressure, administrators can utilize the oom_score_adj interface to manually adjust a process's kill priority.
When these services are isolated within cgroup v2 memory controllers, the interaction between the group-level memory limits and the individual process adjustment values becomes complex. There is uncertainty regarding how the kernel prioritizes a process with a high negative oom_score_adj when the memory limit is reached at the cgroup level rather than the system-wide level.
Does the oom_score_adj value maintain its relative priority across different cgroup hierarchies? What is the recommended value to ensure a service is the absolute last candidate for termination within a constrained cgroup?
34775 reputation · 29 Aug 2023, 10:19 UTC
To ensure a service is the absolute last candidate for termination within a constrained cgroup, set its oom_score_adj value to -1000. This is the minimum possible value and effectively disables the OOM killer for that specific process, provided the kernel can find other candidates to terminate to reclaim memory.
The oom_score_adj value maintains its relative priority across different cgroup hierarchies. When a cgroup v2 memory limit is reached, the kernel identifies the specific cgroup that exceeded its limit and then looks for a victim within that group. The kernel still references the individual process oom_score_adj values to calculate the final kill score. Consequently, a process with -1000 will be bypassed in favor of any process with a higher score residing in the same group.
There is one primary scenario where oom_score_adj is ignored: when the memory.oom.group attribute is enabled for the cgroup. If memory.oom.group is set to 1, the kernel treats the entire cgroup as a single unit. If any process in that group triggers an OOM event, the kernel terminates all processes in that group regardless of their individual adjustment values.
To apply this protection to a running process, write the value to the proc filesystem:
redhatecho -1000 > /proc/[PID]/oom_score_adjTo verify the current protection status and group behavior, use the following commands:
cat /proc/[PID]/oom_score_adj (Should return -1000).cat /sys/fs/cgroup/[cgroup_path]/memory.oom.group (Should return 0 to allow individual process protection).This guidance assumes a standard Linux kernel (e.g., RHEL 8/9) using cgroup v2. Note that setting critical services to -1000 can be dangerous; if every process in a memory-constrained environment is protected, the kernel may be unable to reclaim memory, potentially leading to a system-wide kernel panic or deadlock.
Diagnostic Detail Needed: Is the memory.oom.group attribute enabled for your specific service cgroup? If it is enabled, oom_score_adj will have no effect.
Use comments to ask for clarification. Post a solution as an answer.
6,260 reputation · 29 Aug 2023, 09:06 UTC
One clarification that changes how the protection is applied: there is no memory.oom_score_adj file in cgroup v2. The adjustment is per-task via /proc/[pid]/oom_score_adj and, for systemd units, via OOMScoreAdjust=. The value travels with the task when it moves between cgroups, it does not stay with the cgroup.
That means the protection is set on the service process and its children, not on the controller. The adjustment only reorders victim selection inside the cgroup that hits memory.max; it does not raise the limit, stop reclaim, or prevent OOM pressure. On a RHEL unified hierarchy, confirm cgroup2fs with stat -fc %T /sys/fs/cgroup and check memory.oom.group before assuming per-task scoring applies.