Limiting Service Resources with cgroup v2 and systemd

The Problem: Unbounded Service Resource Consumption

A single malfunctioning service or a sudden spike in traffic can consume all available system memory or CPU cycles, leading to a system-wide freeze or the kernel killing critical processes (the OOM killer). While traditional tools like nice or ulimit provide some control, they lack the precision needed for modern container-like isolation on a bare-metal or VM Linux host.

The solution is Control Groups v2 (cgroup v2). Unlike v1, which had separate hierarchies for memory and CPU, v2 uses a unified hierarchy. This allows systemd to manage all resource controllers for a service in one place, ensuring that memory limits and CPU quotas are applied consistently to the service and all its child processes.

Prerequisites

• Linux Kernel: 4.5 or higher (cgroup v2 support), though 5.0+ is recommended for full stability.
• Init System: systemd version 232 or newer.
• Permissions: Root or sudo access to modify service unit files.

Verifying cgroup v2 Activation

Before applying limits, confirm your system is using the unified hierarchy. Run the following command on the host:

mount | grep cgroup

Expected Result: You should see a line indicating type cgroup2 mounted on /sys/fs/cgroup. If you see multiple mounts for cpuset, memory, and blkio separately, your system is in "hybrid mode" or v1, and the following systemd directives may behave inconsistently.

Implementing Resource Limits

Resource limits are defined within the [Service] section of a systemd unit file. You do not need to manually interact with /sys/fs/cgroup; systemd handles the filesystem writes for you.

1. Memory Constraints

Memory management in cgroup v2 distinguishes between "throttling" (reclaim) and "killing" (OOM).

• MemoryHigh: A "soft" limit. When reached, the kernel aggressively tries to reclaim memory from the process (e.g., clearing page caches). The process is slowed down but not killed.
• MemoryMax: A "hard" limit. If the process exceeds this, the kernel invokes the Out-of-Memory (OOM) killer to terminate the process or its children.

2. CPU Constraints

• CPUWeight: Proportional sharing. If two services both have CPUWeight=100, they split CPU time 50/50. If one has 100 and another 200, the latter gets twice as much time during contention.
• CPUQuota: A hard cap. CPUQuota=50% limits the service to half of one CPU core, regardless of whether the rest of the system is idle.

Example Configuration

To limit a hypothetical data-processor.service, create an override file to avoid modifying the vendor's original unit file:

sudo systemctl edit data-processor.service

Add the following configuration:

[Service]
# Limit memory to 1GB hard cap, start reclaiming at 800MB
MemoryMax=1G
MemoryHigh=800M

# Limit CPU to 2 cores (200%)
CPUQuota=200%

# Give this service lower priority relative to others
CPUWeight=50

Save and exit. Apply the changes with:

sudo systemctl daemon-reload
sudo systemctl restart data-processor.service

Verification and Diagnostics

To verify that the limits are active and being enforced, use the following methods:

Real-time Monitoring

Run systemd-cgtop to see a live view of resource usage grouped by cgroup. This identifies which services are hitting their limits in real-time.

systemd-cgtop

Direct Kernel Inspection

Check the actual memory usage recorded by the kernel for the service:

cat /sys/fs/cgroup/system.slice/data-processor.service/memory.current

The value is returned in bytes. Compare this against your MemoryMax setting.

Comparison: Hard Caps vs. Proportional Weights

Feature	CPUQuota (Hard Cap)	CPUWeight (Proportional)
Behavior	Strict limit; throttles process.	Dynamic; shares based on ratio.
Idle System	Process is still capped.	Process can use 100% of CPU.
Use Case	Preventing a "noisy neighbor".	Prioritizing critical services.

Limitations and Risks

• OOM Risk: Setting MemoryMax too low for a process that doesn't handle memory pressure gracefully will result in immediate SIGKILL crashes. Always set MemoryHigh lower than MemoryMax to give the kernel room to reclaim memory first.
• CPU Throttling: CPUQuota can introduce latency spikes. If a service is time-sensitive (e.g., a database), CPUWeight is generally safer than CPUQuota.

Rollback

To remove the limits, remove the override file and reload the daemon:

sudo rm -rf /etc/systemd/system/data-processor.service.d/
sudo systemctl daemon-reload
sudo systemctl restart data-processor.service