Synchronous vs Asynchronous Cache Invalidation for Feed Consistency

Answer

Use a split invalidation policy rather than one global mode: synchronously acknowledge cache deletion in the write-home region or local quorum before the write returns, then send asynchronous best-effort deletes to remote regions with a short TTL or read-repair fallback. Pure synchronous cross-region invalidation is the only option that minimizes staleness everywhere, but it is the most likely to violate a 10 ms p99 write budget when inter-region delete-ack latency varies. Pure asynchronous invalidation protects write latency but leaves the largest window in which distant readers can receive old rankings.

Likely explanation vs. confirmed constraints

The supplied Memcached/lease details are design context, not guaranteed behavior for your client, mcrouter version, or network. What is confirmed by the question is the constraint set: a 10 ms p99 write budget, variable cross-region replication lag, and lease timeouts that are internal and not exposed for tuning. The likely cause of stale reads after an asynchronous delete is not lease failure; it is that the database write is visible before the remote cache delete arrives, or that a reader hits an entry cached before the delete propagated.

How leases interact during miss storms

Leases bound stale repopulation after a delete: a read miss can receive a lease or token, and a later set is accepted only if that lease is still current. That prevents an old reader from resurrecting a value deleted by a newer write. Leases do not make an already-cached pre-delete entry disappear.

During a cache-miss storm, synchronous local delete-ack serializes invalidation before repopulation in the home region, but it increases tail latency and retry pressure. Asynchronous deletion can overlap with a thundering herd, so deletes must be idempotent and repopulation must read the post-write database value or be lease/CAS guarded.

Per-feed choice

No documented per-feed-type default is established by the supplied context, and lease duration should not be treated as a safe override unless your client exposes it. Choose per feed from measured staleness SLOs and keep the mode behind a feature flag.

Feed profile	Write-home region	Remote regions	Staleness control
Marketplace-like, tight read-after-write SLO	Synchronous delete-ack to local quorum	Async delete plus short TTL	Read-after-write checks and read repair
News Feed-like, tolerant of brief remote staleness	Async or sync local if budget allows	Async best-effort delete	Short TTL plus lease/CAS repopulation

Verification before rollout

1. Measure write p99 with invalidation disabled, then with local-quorum delete-ack, then with cross-region fan-out. Reject synchronous cross-region if delete-ack p99 approaches or exceeds 10 ms.
2. Run read-after-write checks from clients in each region after a feed write and record the maximum observed staleness against the per-feed SLO.
3. Replay a cache-miss storm in staging and verify stale sets are rejected by lease/CAS and duplicate or retried deletes remain idempotent.
4. Roll out synchronous local invalidation first for tight-SLO feeds; enable async remote only where measured staleness is acceptable.

# Scoped staging check only: one feed ID, one region pair.
# Record p99 for: baseline write, local quorum delete-ack,
# cross-region fan-out, then read-after-write staleness per region.

The one diagnostic that would change this recommendation is measured delete-ack p99 for local quorum versus cross-region fan-out during failover. If local quorum acknowledgment is already near the budget, move even local invalidation to async plus explicit read-after-write repair.