Diagnosing TAO Read Latency Spikes: Cache Miss Storms and Hot Object Amplification

Recognizable Condition

A sudden increase in p99 and p999 latency for TAO-backed reads—friend lists, profile views, and feed fan-out—while writes remain normal is the signature of a serving-path read problem. Operators typically see elevated database QPS, increased memcached misses, and user-visible partial loads or timeouts. The condition is cache-driven when latency spikes coincide with a drop in cache hit rate rather than a change in write volume.

Cause and Diagnostic Table

Observation	Likely Cause	Diagnostic Signal
Hit rate collapses across many keys after restart or invalidation	Cache miss storm from cold restart or bulk invalidation	DB QPS rises sharply; memcached miss rate jumps; per-key latency uniform
Latency concentrated on specific popular objects	Hot key or popular user fan-out creating request skew	Key heatmap shows skewed distribution; fan-out QPS exceeds baseline
Gradual hit-rate decline under steady traffic	Memcached eviction pressure	High slab utilization; rising eviction rate; connection errors
Stale reads followed by client retries	Replica lag on TAO backing tables	MySQL replica lag metric increases; retry rate amplifies DB load
Tail latency with cache timeouts	Network partition between serving tier and cache	Connection errors to memcached; increased timeout counts

Ordered Checks

1. Confirm TAO hit/miss ratio and per-key latency. Compare the current hit rate against the same time window on previous days. A sharp drop points to cache-driven amplification rather than organic traffic growth.
2. Inspect memcached health. Check eviction rate, slab utilization, and connection errors. High eviction with full slabs indicates memory pressure; connection errors suggest network or capacity issues.
3. Check MySQL replica lag and QPS. For TAO backing tables, verify whether replica lag is rising and whether QPS has spiked. Rising QPS with falling hit rate confirms a miss storm.
4. Sample hot keys. Use the key heatmap and request distribution view to identify whether a small set of keys dominates traffic. This distinguishes a general miss storm from targeted fan-out amplification.
5. Review recent changes. Correlate the latency onset with deployments, invalidations, or traffic shifts in release and operational logs.

Fixes Tied to Findings

Cache miss storm

If checks show a broad hit-rate collapse after a cold restart or bulk invalidation, enable TAO warming to repopulate hot objects before traffic returns. Temporarily throttle fan-out and increase read replica capacity to absorb database load while the cache recovers.

Hot key amplification

If the heatmap shows skewed request distribution around popular users or objects, apply key splitting to distribute load across cache nodes. Add read replicas or rate limit fan-out for the affected objects. Validate that downstream write amplification and storage cost remain acceptable before making the change permanent.

Eviction pressure

When slab utilization is high and eviction rates are climbing, increase memcached capacity, adjust TTL values, or reduce object size. Avoid flushing caches or restarting serving tiers during peak traffic, as this can trigger a wider miss storm.

Replica lag

If replica lag is causing stale reads and retry storms, shed non-critical reads from the lagging replica and promote a healthy replica if the lag persists.

Network partition

Connection errors between the serving tier and cache indicate a network partition. Escalate to infrastructure owners to restore connectivity; do not attempt cache flushes while partitions are unresolved.

Escalation Criteria

• p99 latency exceeds baseline for more than 5 minutes with a rising error rate
• Database CPU sustained above 80 percent
• Cache hit rate drops more than 15 percent below baseline
• Hot key QPS exceeds documented safe fan-out limits

Limitations and Verification

This guide assumes the TAO architecture as publicly described in the 2010s to early 2020s. Internal service names, thresholds, and topology vary by region and product and may have evolved. Treat thresholds as starting points for your own baselines.

To verify a fix, compare TAO hit rate trends with memcached miss rate and database QPS on the same time window to confirm cache-driven amplification has stopped. Correlate latency spikes with deployment, invalidation, or traffic shift events in release and operational logs. Validate recovery by observing hit rate recovery and latency decay over 10 to 15 minutes, and confirm the error rate returns to baseline.