Taming Memory Spikes with Linux cgroups v2: Beyond the Hard Limit

The OOM Killer is a Blunt Instrument

When a process exceeds its allocated memory in a containerized environment, the Linux kernel usually has one primary response: the Out-of-Memory (OOM) Killer. It identifies the process consuming the most memory and terminates it immediately to save the rest of the system. For a production service, this is a catastrophic failure—a hard crash that drops connections and clears caches.

The problem is that most engineers only use memory.max (the hard limit). This creates a binary state: the process is either healthy or dead. To build resilient systems, you need a way to signal the kernel to start reclaiming memory before the process hits the wall. This is where cgroups v2 (Control Groups version 2) and its tiered memory controls become essential.

The Unified Hierarchy Advantage

In cgroups v1, memory and CPU were managed in separate trees. This fragmentation often led to "controller overlap," where a process was tracked differently by different subsystems, making resource accounting a nightmare. cgroups v2 introduces a unified hierarchy. Every process belongs to exactly one group, and all controllers (memory, CPU, I/O) are applied to that same group.

This unification allows the kernel to make smarter decisions. For example, it can coordinate memory reclaim efforts with CPU pressure, ensuring that the system doesn't waste CPU cycles trying to reclaim memory that is currently being actively written to disk.

Three Tiers of Memory Control

Instead of a single limit, cgroups v2 provides three distinct levers to manage memory pressure:

• memory.low (The Floor): This is a "best-effort" guarantee. The kernel will avoid reclaiming memory from a group unless the rest of the system is under significant pressure. It prevents a low-priority background task from stealing the page cache of your primary application.
• memory.high (The Throttle): This is a soft limit. When a group exceeds this value, the kernel aggressively attempts to reclaim memory (e.g., by clearing the page cache). If the process continues to allocate, the kernel will slow down the process's allocations to give the reclaim logic time to work. The process keeps running, but it slows down.
• memory.max (The Ceiling): The hard limit. If the process hits this and the kernel cannot reclaim enough memory to stay under the limit, the OOM Killer is invoked.

Practical Example: Setting a Memory Buffer

Imagine a service that normally uses 512MB but occasionally spikes to 1GB during heavy requests. You want to guarantee 256MB, warn the kernel at 768MB, and kill it only if it hits 1.2GB.

Prerequisites: You must be root or have sudo permissions. This example assumes a kernel version 4.5+ and a system using cgroups v2.

1. Verify you are using cgroups v2:

# Run this on the host machine
stat -fc %T /sys/fs/cgroup
# Expected output: cgroup2fs

2. Create a group and apply limits:

# Create the group directory
sudo mkdir /sys/fs/cgroup/my_service

# Set the floor (256MB)
sudo bash -c "echo 256M > /sys/fs/cgroup/my_service/memory.low"

# Set the soft limit (768MB)
sudo bash -c "echo 768M > /sys/fs/cgroup/my_service/memory.high"

# Set the hard limit (1.2GB)
sudo bash -c "echo 1.2G > /sys/fs/cgroup/my_service/memory.max"

3. Assign a process to the group:

# Move a running process (PID 1234) into the group
sudo bash -c "echo 1234 > /sys/fs/cgroup/my_service/cgroup.procs"

4. Monitor the result:

# Check current usage
cat /sys/fs/cgroup/my_service/memory.current

Trade-offs and Limitations

While memory.low provides a safety net, it is not a hard reservation. If the entire system is running out of memory, the kernel will ignore memory.low and reclaim memory from any group to prevent a total system kernel panic. Over-provisioning memory.low across too many groups can lead to "memory fragmentation" where the kernel cannot find enough contiguous pages for critical system tasks.

Additionally, some legacy container runtimes (older versions of Docker or containerd) may still rely on cgroups v1. If your environment is hybrid, you may find that these settings are ignored or cause the container to fail to start.

Verification and Rollback

To verify the settings are active, monitor memory.events in the group directory. If you see the high counter incrementing, the kernel is actively throttling the process to keep it under the soft limit without killing it.

Rollback: To remove the limits and the group, move the processes back to the root group and delete the directory:

sudo bash -c "echo 1234 > /sys/fs/cgroup/cgroup.procs"
sudo rmdir /sys/fs/cgroup/my_service