Memcached eviction limits under per-slab LRU pressure

Acceptance criteria

Use per-class counters, not aggregate memory utilization. For each slab class, track:

• Occupancy: used_chunks / total_chunks from stats slabs. Flag any class approaching 100 percent, even when other classes have free chunks.
• Live evictions: the evictions counter. Nonzero live evictions during steady load mean the class is evicting least-recently-used items to make room.
• Expired-slot reuse versus live eviction: compare reclaimed with evictions. reclaimed counts reuse of expired slots and is not evidence of live pressure.
• Out-of-memory: the outofmemory counter for rejected stores.

Confirmed behavior and assumptions

Confirmed: memcached divides memory into slab classes, and each class keeps an independent LRU. When a class needs a chunk, it evicts from its own tail; free memory in another class does not prevent that eviction. There is no global LRU ordering and no built-in hot-pin guarantee. Default limits are a 1 MB maximum item size and a 250-byte maximum key length, both configurable at startup; oversized items are rejected rather than stored.

Assumption to verify: counter names and availability vary by memcached build and flags. Treat the research brief as planning context, not verified evidence for your binary. Check the deployed node's stats output before finalizing thresholds.

Verification steps

1. Run stats slabs on each node and compute per-class occupancy. Identify classes near full.
2. Run stats and poll evictions, reclaimed, and outofmemory. Investigate any class where evictions rises while reclaimed stays flat.
3. If your build exposes per-class item stats, confirm which class owns the hot keys before setting class-specific limits.
4. Optionally, in a staging cluster, fill one slab class while leaving others empty and verify that evictions appear only in the full class.

Trade-off: retention versus utilization

Yes. If the SLA is hot-key retention, prioritize preventing hot-key eviction over maximizing aggregate utilization. Measure the trade-off with:

• Hot-key hit ratio for keys in the pressured classes.
• Per-class early-eviction rate: live evictions of TTL-bearing items before expiry.

Accept the policy only if hot classes show zero or near-zero early evictions. Aggregate utilization can stay low in other classes; that is expected under per-class LRU isolation. If hot keys span many classes, memcached cannot guarantee retention by itself; use application-level sharding by item size, over-provisioning, or a different cache layer.

Missing diagnostic detail

Is slab automove enabled on the deployed nodes? If automove is active, pages migrate between classes and per-class occupancy is not stable, which changes whether fixed occupancy thresholds are meaningful.