LRU Crawler vs Lazy Expiration: Choosing Reclamation Strategy Under High Write Throughput

Question

When memcached runs under sustained high write throughput, expired items can accumulate and either waste memory or trigger premature evictions. Two documented reclamation paths exist: lazy expiration, which checks expiry only on item access, and the LRU crawler, a background thread that proactively scans and removes expired entries.

Enabling the crawler (-o lru_crawler) reduces the window that expired items occupy memory but adds CPU cycles and can increase contention on the global LRU locks, especially on many‑core systems. Lazy expiration avoids background overhead but lets stale items linger, potentially causing valid entries to be evicted under memory pressure.

A further complication arises when slab automove (-o slab_automove) reassigns pages between slab classes while the crawler is iterating; the exact interaction is not fully detailed in the official docs, making the combined behavior uncertain.

Which reclamation approach—or combination of tunables such as lru_crawler_sleep and lru_crawler_tocrawl—provides the best trade‑off for a workload with heavy writes and active slab automove? How can the crawler’s impact on LRU lock contention be measured in practice? Does the current source code guarantee safe page moves during a crawler scan?

Tasadduq Burney · Answer

Direct answer
For sustained high write throughput with slab_automove active, use lazy expiration as the baseline and add the LRU crawler only as a measured, conservative supplement. Do not enable crawler and slab_automove together by default; their interaction is version-dependent and not clearly guaranteed safe in public documentation. Start with the crawler off, measure under load, then enable it with a non-zero lru_crawler_sleep and a small lru_crawler_tocrawl while keeping slab_automove disabled or fixed until lock contention is quantified.
Confirmed behavior versus likely explanation
Confirmed: lazy expiration removes an expired item only when that key is accessed or when the LRU selects it for eviction. Under write-heavy workloads with few reads of stale keys, dead items can remain resident and push live entries out prematurely. Confirmed: the crawler proactively deletes expired items, shortening stale-memory residency, but it holds item/LRU locks while scanning and adds CPU. Confirmed: slab_automove reassigns pages between slab classes based on eviction pressure; whether a page can move while the crawler iterates it depends on the memcached release.
Likely explanation: high write rates fill slab pages quickly. Expired but unread items sit near the LRU tail, so evictions for new writes may discard live data before expired data is reclaimed. The crawler reduces that window but competes with gets and sets for the same LRU locks; on many-core hosts this can serialize item access and lower throughput. slab_automove page moves during a scan can invalidate the crawler's position or force rescan, adding uncertainty rather than predictable reclamation.
Steps for this case

Run the production-like write workload with the crawler off and slab_automove in its current state. Record stats items fields evicted, expired, and reclaimed, plus get/set latency and CPU.
Enable the crawler with conservative settings: non-zero lru_crawler_sleep and small lru_crawler_tocrawl. Keep slab_automove off or pinned if your deployment allows. Repeat the identical load and compare eviction rate, expired/reclaimed counts, CPU delta, and tail latency.
If evictions fall without meaningful CPU or latency regression, keep the crawler. If lock wait or tail latency rises, increase lru_crawler_sleep or reduce lru_crawler_tocrawl.
Re-enable slab_automove only after crawler behavior is stable, and validate the exact memcached release for crawler-plus-automove interaction before production rollout.

Measuring crawler lock impact
Toggle -o lru_crawler while holding write throughput constant. Use stats items to compare evicted, expired, and reclaimed across runs. Profile memcached with perf top or lockstat during crawler passes and look for time spent in LRU/item mutexes; compare get and set tail latency with crawler on and off. Sample stats slabs around crawler cycles to detect page-class changes while the crawler is active.
Missing diagnostic detail
Provide the exact memcached version. Crawler-plus-automove safety is version-dependent, and that detail determines whether combining them is acceptable or should be avoided entirely.
Version assumption: this guidance assumes a recent memcached 1.6.x line; crawler and automove locking differ across releases. No public source-code guarantee of safe page moves during an active crawler scan is assumed; treat that interaction as unverified until tested on your exact release.

LRU Crawler vs Lazy Expiration: Choosing Reclamation Strategy Under High Write Throughput

1 answer

Direct answer

Confirmed behavior versus likely explanation

Steps for this case

Measuring crawler lock impact

Missing diagnostic detail

Your perspective belongs here.

0 question comments