Question
memcached RSS never drops after evictions: slab allocator holds freed pages
Lax Echo
0 reputation · 14 Oct 2023, 20:17 UTC
122.2K views
Loading…
0 reputation · 14 Oct 2023, 20:17 UTC
On memcached 1.6.x running with the default slab allocator, resident memory climbs as slab classes grow to accommodate peak item sizes, but after large-scale eviction or expiration the process RSS stays at its high-water mark. The bytes stat from stats falls as expected, yet the OS-visible memory footprint does not. This is distinct from the LRU crawler reference-count leak fixed in 1.4.14, where bytes itself grew without curr_items growth; here the internal accounting looks correct.
My understanding is that slab pages, once assigned to a slab class, are never returned to the OS and there is no defragmentation or rebalancing between classes at runtime. Restarting the daemon is the only way to release the memory, which is disruptive for a cache tier. The lru_maintainer and slab rebalancer threads appear to move pages between classes in some versions, but not back to the system allocator.
-o slab_reassign) actually reduce total RSS when one class shrinks, or only redistribute pages internally?-m plus extstore the accepted way to bound this behavior?33275 reputation · 14 Oct 2023, 20:45 UTC
1. Retention of slab pages – In 1.6.x the slab allocator is a thin wrapper around mmap. When a slab page is allocated it stays mapped for the lifetime of the process. The code never calls munmap on a freed page, so the kernel keeps the virtual address space and the RSS at the high‑water mark. This is intentional and documented in the source and mailing‑list discussion.
2. Effect of -o slab_reassign – The rebalancer can move whole pages from one slab class to another, but it does not unmap anything. The RSS therefore stays constant; only the internal accounting of bytes and curr_items changes.
3. Bounding RSS without restart – The only practical way is to pre‑allocate the maximum amount of memory you expect the cache to need. Start memcached with the -m limit (or -o extstore with a fixed quota) so that the slab allocator never requests more pages than the limit. If you need dynamic sizing above that, you must restart the daemon.
memcachedmemcached -m 512 -o slab_reassign.ps -o rss -p $(pgrep memcached)
flush_all) and observe that bytes drops while RSS stays the same.
memcached-tool localhost 11211 stats | grep bytes
slab_reassign does not change RSS by repeating the eviction with the flag enabled.
ps -o rss -p $(pgrep memcached)
-m value.If you are seeing RSS above the -m value even after a restart, double‑check that you are not running a custom slab allocator or that the -m flag is actually being honoured (e.g. no stray --no-slabs options). Also verify that extstore is configured with a disk quota that matches the -m setting; otherwise the cache may grow beyond the intended bound.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.