Question
cgroups v2 Unified Hierarchy: Resource Distribution Ambiguity in Internal Nodes
Zo Nea
0 reputation · 07 Apr 2021, 11:45 UTC
71.8K views
Loading…
0 reputation · 07 Apr 2021, 11:45 UTC
Linux cgroups v2 utilizes a unified hierarchy to ensure a process belongs to exactly one group across all controllers. To prevent ambiguity in resource distribution, the kernel enforces a "no internal process" rule, which prohibits processes from residing in a non-leaf node if that node also contains child groups.
When managing complex service trees, there is a technical tension between the desire to apply broad resource limits at a parent level and the requirement to move processes into child groups for granular control. If a process is active in a parent group, attempting to create or move processes into a subgroup can trigger errors, as the kernel cannot determine whether the parent's resources should be shared with the child or reserved for the internal process.
Given this architecture, how does the kernel resolve the priority of resource allocation when a leaf node is created in a hierarchy that previously held the only active processes? Does the transition of processes from an internal node to a leaf node impact the immediate enforcement of memory.high or cpu.weight for sibling groups?
34775 reputation · 07 Apr 2021, 16:43 UTC
The kernel does not arbitrate priority between an internal process and its children. In cgroups v2 unified hierarchy the "no internal process" rule makes the state illegal: a cgroup that has descendant cgroups must not contain tasks. When a leaf is created under a parent that previously held the only active processes, resource distribution for siblings is not active until the parent becomes a pure internal node. memory.high and cpu.weight on the parent can only be applied to the set of children once the parent has zero tasks. The transition of the last task from the parent to a leaf makes the parent eligible for distribution, and the controllers then account usage and enforce limits across the children immediately from that point.
linuxcgroup.procs for a non-leaf cgroup is rejected.memory.max are hard ceilings for the sum of all descendants. memory.high is a soft pressure threshold that triggers reclaim/throttling within the subtree. cpu.weight defines proportional share among siblings under a common parent.Creation of the leaf alone does not move processes and does not change enforcement. The kernel does not automatically migrate tasks. Until the administrator moves all tasks out of the parent, the parent remains a leaf in practice and the distribution algorithm is not used for its children. Once the last task is removed, the parent is reclassified as internal and the kernel can apply cpu.weight sharing and memory.high pressure handling across its children. The effect on existing sibling groups is immediate after the parent becomes task-free, because accounting switches from per-cgroup to hierarchical aggregation.
cat /sys/fs/cgroup/cgroup.controllers should list controllers.cat /sys/fs/cgroup//cgroup.procs./sys/fs/cgroup//cgroup.procs. Repeat until parent cgroup.procs is empty.cat /sys/fs/cgroup//cgroup.procs is empty and children exist under /sys/fs/cgroup//.cat /sys/fs/cgroup//memory.current and cpu.weight now reflect aggregation and distribution across children.One diagnostic detail that changes the recommendation: is the hierarchy managed by systemd? If systemd owns the tree, manual moves may be reverted and limits should be set via systemctl set-property or slice units instead of direct writes.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 07 Apr 2021, 20:31 UTC
When a task moves from an internal cgroup to a newly created leaf child, the kernel instantly treats the parent as empty of direct tasks. The parent’s cpu.weight is then divided solely among its child cgroups, so sibling leaf groups receive the full weight without any scheduling delay. Likewise, memory.high remains a ceiling on the combined usage of the parent and its descendants; freeing memory in the parent lowers actual consumption but does not change the limit, allowing children to use up to the same threshold immediately. You can verify this by watching cpu.stat or memory.stat in the child cgroups after the move – the values update on the next scheduler tick or page‑reclaim cycle.