Diagnosing Memcached Slab Fragmentation and Eviction in Production

When Memcached evicts but memory looks free

Memcached does not manage memory as a single pool. At startup it divides its allocation area into slab classes—predefined size buckets—and every stored item must fit into a chunk from exactly one class. When your actual item sizes do not align well with those buckets, memory becomes stranded: one slab class runs out of free chunks while another holds mostly empty pages. The result is a confusing production symptom: evictions climb, clients see out-of-memory errors, yet stats still reports free bytes.

The useful takeaway is that you should treat slab layout as a first-class diagnostic target. Before adding nodes or raising the memory limit, confirm whether your slab classes match your workload. Runtime tuning can relieve crawler and thread contention, but correcting a fundamental size mismatch requires a restart.

Recognizable conditions

• Rising evictions with low memory utilization. stats shows plenty of free bytes, but evictions increases steadily under normal load.
• Out-of-memory errors despite reported free space. Clients receive allocation failures even though total memory use appears moderate.
• Get latency spikes during rebalancing or expiry sweeps. Latency jumps coincide with background LRU crawler activity rather than request volume.
• Connection timeouts under moderate load. Accepts succeed, but request processing stalls when concurrency rises.

Cause and diagnostic mapping

Symptom	Likely cause	Diagnostic signal
Rising evictions, low memory use	Slab class mismatch	stats slabs shows free_chunks=0 in active classes while other classes retain large free capacity
OOM errors despite free bytes	Memory fragmented across classes	No single slab class has contiguous chunks sized for the incoming item
Get latency spikes during rebalancing	LRU crawler contention	Crawler active on busy slabs; lru_crawler_sleep too aggressive for the workload
Connection timeouts under moderate load	Thread pool exhaustion	Default worker threads insufficient for long-running SASL auth or large multiget parsing

Ordered verification steps

Run the following checks from a monitoring host or bastion with network reachability to the Memcached port (default 11211). No special OS permissions are required; any client that can open a TCP connection can issue these text-protocol commands.

Step 1 — Inspect slab class occupancy

Use memcached-tool or raw protocol to dump slab statistics:

memcached-tool <host>:<port> stats slabs

Alternatively, if memcached-tool is unavailable:

printf "stats slabs\r\n" | nc <host> <port>

Expected check: identify slab classes where free_chunks is zero and eviction counters are climbing, then compare them with classes holding more than 50% free chunks. A concrete decision point: if class 3 is exhausted while class 8 sits mostly empty, your incoming items likely fall between the chunk sizes those classes provide. Note that slab statistics reflect an instantaneous state; capture a series across a full load cycle before concluding that fragmentation is persistent.

Step 2 — Correlate operations with evictions

printf "stats\r\n" | nc <host> <port>

Compare cmd_get, cmd_set, and evictions. If sets are high and evictions track them proportionally, your working set simply exceeds the configured memory limit. If evictions remain high while set volume is moderate, suspect slab misalignment rather than raw capacity.

Step 3 — Check allocator and crawler settings

printf "stats settings\r\n" | nc <host> <port>

Verify slab_page_size (commonly 1MB), chunk_size_growth_factor (default 1.25), and whether the LRU crawler is enabled. Memcached 1.6 and later enable the crawler by default; older versions require the -o lru_crawler startup flag.

Targeted fixes tied to findings

Fix slab class mismatch (restart required)

If Step 1 shows stranded free chunks in the wrong classes, adjust the growth factor and minimum chunk size at startup:

memcached -m <mb> -f <factor> -n <min_bytes> -p <port> -u <user>

The -f flag controls how quickly chunk sizes grow between classes; the -n flag sets the smallest chunk size. For example, if your items cluster tightly around a size that falls into a gap created by the default 1.25 factor, a smaller factor such as 1.1 may reduce wasted space, though it also increases the number of slab classes and per-class metadata overhead. This change requires a full restart and invalidates the existing cache, so plan a maintenance window. Rollback is to revert the startup flags and restart again.

Reduce LRU crawler contention (runtime)

If latency spikes correlate with crawler activity, increase the crawler sleep interval to reduce background CPU pressure:

printf "config set lru_crawler_sleep 500\r\n" | nc <host> <port>

The value is in microseconds. Raising it lowers CPU contention but delays reclaim of expired items; setting it too low steals cycles from worker threads. This is a runtime change, so you can revert it with another config set without restarting.

Relieve thread starvation (restart required)

If connection timeouts appear under moderate load, increase worker threads and listen backlog:

memcached -t <threads> -b <backlog> -p <port> -u <user>

A common starting point is two worker threads per CPU core. Raising -t increases per-connection memory overhead, so monitor total RSS against your limit. Rollback requires reverting the flags and restarting.

Escalation criteria

Escalate to a capacity or architecture review when any of the following thresholds are crossed:

• Eviction rate exceeds 10% of cmd_set sustained for 15 minutes.
• p99 get latency exceeds 50ms for 5 consecutive minutes.
• Out-of-memory errors exceed 1 per second.
• Slab class fragmentation remains above 70% after growth-factor tuning.

Limitations and practical verification

Slab statistics are instantaneous snapshots. A single reading during a quiet period can look healthy while fragmentation builds under peak load. Capture stats slabs repeatedly across a representative traffic window before changing startup flags.

To verify a fix, compare memcached-tool <host>:<port> stats slabs before and after the change, confirming that free chunks are distributed across the classes your workload actually uses. For a more rigorous check, replay a representative workload with a load generator such as memaslap and measure eviction rate and p99 latency over a 30-minute window. If you enabled verbose logging with -vv, temporarily observe which slab classes sample keys are assigned to, then disable it. After adjusting the listen backlog, watch stats conns to confirm connection yield counts drop.

Remember that growth-factor tuning trades memory efficiency for class granularity. More classes reduce waste but increase bookkeeping, and the change is not reversible without another restart. Runtime crawler and backlog adjustments are safer first steps, but they cannot fix a fundamental mismatch between item sizes and slab layout.