SERVER_ERROR out of memory in memcached slab allocator

Question

SERVER_ERROR out of memory in slab allocator

Memcached's slab allocator divides memory into fixed-size chunks grouped into slab classes. Once a page is assigned to a specific slab class it cannot be reassigned to another class during the daemon's lifecycle.

A shift in stored item sizes can exhaust one slab class while other classes retain free memory. In that state the server returns SERVER_ERROR out of memory to clients even though total available memory is sufficient for the request. Pre-allocation with -L and growth factor -f only define startup behavior and cannot adapt to runtime workload changes. No native mechanism exists to move pages between classes or trigger global defragmentation without a restart.

Runtime constraints

The unresolved design decision is whether page rebalancing can be implemented to allow dynamic slab class resizing without downtime, a topic discussed but not adopted due to complexity and lock contention concerns.

Is a documented method available to force reallocation of pages from an underutilized slab class to one reporting SERVER_ERROR out of memory?
Which runtime signals indicate slab class imbalance before client errors occur?
Can the daemon maintain cache contents while correcting slab class exhaustion?

Tasadduq Burney · Answer

Short answerThere is no documented way to force an arbitrary page from an underutilized slab class into an exhausted one on demand. Memcached assigns each 1 MB page to exactly one slab class for the life of the daemon, so SERVER_ERROR out of memory means the class matching your item size is full — not that the server is out of memory overall. The only complete reset of the slab layout is a restart, which discards the cache. Newer versions (1.4.24+) add partial mitigations, described below.What is confirmed vs. likelyConfirmed behavior: pages are permanently bound to a slab class at assignment time; -f (growth factor) and -L (preallocate/huge pages) only shape startup layout; aggregate free memory does not reflect per-class exhaustion.Likely explanation for your case: your item size distribution shifted (e.g., values grew slightly and crossed a class boundary), so one class filled while neighbors sit idle. This is the classic cause of OOM errors with plenty of RAM free.Partial runtime mitigation: slab_automoveSince memcached 1.4.24, the daemon supports -o slab_automove=1 (and slab_automove=2 for a more aggressive mode). When a class is about to evict or fail, the daemon may reassign a page from another class. Caveats:It is heuristic and triggered under pressure, not an on-demand rebalance command. There is no slab reassign -style API you can call yourself.Pages being moved must be emptied first, which can itself cause evictions in the source class.It must be enabled at startup; you cannot turn it on for a running daemon.Check your version with memcached -h or stats output — if you are on 1.4.24 or later and did not enable automove, that is the first change to make.Signals that warn you before clients see errorsWatch per-class data, not totals:echo "stats slabs" | nc 127.0.0.1 11211Key fields per class: used_chunks vs total_chunks (a ratio pinned near 1.0 means the class is exhausted), mem_requested vs total_pages * chunk_size * chunks_per_page, and evicted / outofmemory counters. A nonzero, rising outofmemory counter on one class while others show free chunks confirms imbalance. Alert on used_chunks / total_chunks > 0.9 per class rather than on overall memory.Can you fix it without losing cache contents?Not fully. A restart is the only way to rebuild the slab layout from scratch, and memcached has no persistence — a restart is a cold cache. To limit impact:Enable slab_automove at next restart so the layout adapts going forward.Tune -f so class boundaries better match your actual item sizes (inspect stats items size distribution first).If the service cannot tolerate a cold cache, run a second memcached instance, shift traffic to it, restart the first, then shift back — a rolling restart instead of a hard one.One detail that would change this adviceWhich memcached version are you running? If it predates 1.4.24, upgrading is the single most effective step; if you are already current with automove enabled and still hitting this, the real fix is adjusting -f or splitting workloads across instances by item size.

SERVER_ERROR out of memory in memcached slab allocator

SERVER_ERROR out of memory in slab allocator

Runtime constraints

1 answer

Short answer

What is confirmed vs. likely

Partial runtime mitigation: slab_automove

Signals that warn you before clients see errors

Can you fix it without losing cache contents?

One detail that would change this advice

Your perspective belongs here.

0 question comments