Diagnosing Cache Stampede in a Memcached-backed Read Path
A diagnostic guide for recognizing and mitigating cache stampede in a Memcached-backed read path. Covers correlated miss and DB spikes, lease and early recompute checks, and escalation criteria.
16 Oct 2026, 17:26 UTC
Recognizable condition
You see periodic latency spikes on read-heavy social graph paths and a simultaneous jump in backend database QPS. The spikes line up with cache misses for a small set of high-traffic keys, not with traffic growth. Hit rate recovers quickly after the spike, which points to a thundering herd, also called a cache stampede.
A thundering herd happens when a hot cache key expires and many workers miss at once. Each worker then fetches the same data from the origin, amplifying load. In a Facebook-style deployment with a large distributed Memcached tier fronting read-heavy queries, this pattern is recognizable by correlated metrics: cache miss rate uptick for a specific key pattern, backend request queue growth, and p99 latency increase for the affected endpoint while overall request rate is flat.
Cause and diagnostic table
| Observed signal | Likely cause | Why it fits |
|---|---|---|
| Spike in misses for one key prefix, e.g. profile:feed:{user_id} | Key expiration without lease protection | All workers see miss at same TTL boundary |
| Misses spread across many keys at once | Node loss with consistent hashing remap | Consistent hashing limits reshuffle but does not eliminate cold data after a node failure |
| Latency spike but DB QPS flat | Cache server queuing or network saturation | Stampede not reaching origin |
| Repeated spikes after deploy | Cold start, no cache warming | New instances start with empty slabs |
Ordered checks
1. Confirm key hotness and expiration alignment
From your metrics store, inspect miss rate and latency per key pattern over the last window. Look for a sharp, narrow miss burst followed by rapid recovery. Check TTL distribution for the pattern. If many keys share the same TTL and are set by the same writer, expiration alignment is likely.
Where to run: metrics query UI or read-only analytics role. Risk: none. Expected check: identify a small set of keys responsible for the spike.
2. Inspect cache access path for lease or recompute guard
Review the read path code for the affected service. A lease mechanism allows one worker to become the recompute owner while others serve stale data or wait. Probabilistic early recomputation refreshes the value before TTL expiry to avoid a hard boundary.
Where to run: code repo with read access. Risk: none. Expected check: presence of a lease acquisition step on miss, and a fallback to stale value when lease is held by another worker.
3. Check cache topology health
Verify Memcached node membership and consistent hash ring stability. A node flap causes key remapping and temporary misses. Consistent hashing reduces reshuffle scope but does not eliminate impact on hit rate.
Where to run: operations dashboard for cache fleet. Required permission: read-only fleet view. Expected check: no recent node removals, evictions, or slab churn coinciding with the spike. Risk: none.
4. Validate backend capacity headroom
Compare backend QPS and queue depth during the spike to baseline. Over-reliance on caching can mask database regressions until a stampede forces real load.
Where to run: DB monitoring with read access. Expected check: queue growth correlates with cache miss burst. Risk: none.
5. Review deployment and warming
Check recent deploys for the serving tier or cache tier. Cold start without cache warming produces latency spikes as the first requests populate the cache.
Where to run: deploy logs with read access. Expected check: deploy timestamp aligns with spike onset.
Fixes tied to findings
Expiration alignment found
Add jitter to TTL or enable probabilistic early recomputation so keys are refreshed before expiry. If a lease is already present, ensure the lease TTL is longer than the recompute path and that stale reads are allowed during recompute. Lease mechanisms add complexity and can increase latency for the winning request, but they prevent herd.
Missing lease guard
Introduce a lease on cache miss: first worker acquires lease, recomputes, writes back; others return stale value or wait briefly. Do not change TTL alone without a guard, as it can shift the herd.
Node loss or remap
Avoid immediate traffic shift to a new node set. Allow warm-up and monitor hit rate per vnode. Consistent hashing reduces reshuffle but you still need to observe cold key impact.
Cold start after deploy
Apply cache warming for hot key patterns before cutting traffic. Warming prevents the first wave of requests from hitting the origin.
Limitations and verification
Lease and early recompute add request handling complexity and can briefly increase latency for the recompute holder. Caching can mask underlying database performance regressions until a flush or stampede occurs.
Practical verification: after a change, watch the same key pattern for a full TTL cycle. A successful mitigation shows miss bursts flattening and backend QPS staying near baseline during the window where expiry previously occurred. Do not rely on average hit rate alone; inspect per-key miss distribution.
Escalation criteria
Escalate to database on-call if backend queue depth exceeds safe thresholds during a stampede, or if lease contention causes repeated timeouts. Escalate to cache platform if node flaps recur or consistent hash ring changes are frequent. Escalate to service owners if hot key patterns lack lease protection and cannot be warmed safely.
facebook0 replies
A thoughtful contribution can make all the difference. Be the first to share one.