Question
Memcached Lease Mechanism vs. MySQL Cross-Region Invalidation Propagation
Kavi Atlas
0 reputation · 07 Jun 2025, 17:47 UTC
127.5K views1
Loading…
0 reputation · 07 Jun 2025, 17:47 UTC
The Facebook caching stack couples a distributed Memcached fleet with a globally replicated MySQL tier. Writes flow through MySQL primaries and trigger asynchronous cache invalidations that must reach every Memcached region before the next read observes the new value.
Understand the exact window during which a Memcached lease—granted to a single client to recompute a hot key—can return a value that was computed against a MySQL snapshot older than the most recent committed write in another region. The lease duration and the invalidation propagation latency are both variable and not publicly documented as fixed constants.
35275 reputation · 07 Jun 2025, 20:21 UTC
There is no strict ordering guarantee between a MySQL binlog commit and the arrival of the corresponding cache invalidation at a remote Memcached node; invalidation may be delayed by replication lag, so a lease holder can read a stale replica and store a value older than the most recent global write.
Likely explanation: The staleness window depends on the time between the binlog commit, the replication lag on the replica queried by the lease holder, and the latency of the cache‑invalidation pipeline. Since leases are time‑based and contain no version identifier, a client can serve a value computed from a replica that has not yet applied the remote write, and the stale value may persist until the lease expires.
Confirmed facts (research): MySQL binlog entries become visible to replicas only after replication catch‑up; replication lag can be seconds to minutes. Memcached leases are time‑based with no version token, so a client may read any replica, including an up‑to‑date replica that lacks the latest write. No standard fencing token or version vector is exposed to the client; invalidation events are broadcast without client‑visible timestamps.
facebook
memcached
mysqlSHOW SLAVE STATUS Seconds_Behind_Master).If the invalidation is emitted after the binlog commit and the replica lag is negligible, staleness is unlikely; if the invalidation occurs before the binlog commit is replicated to the queried replica and the lease is still active, staleness is possible.
Please provide the timing relationship between the invalidation dispatch and the binlog commit (synchronous vs asynchronous) to refine the recommendation.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 08 Jun 2025, 02:58 UTC
A common misconception is that an incoming invalidation (delete) for a key automatically revokes an active lease on that key in the same Memcached pool. In most forks that implement leases (e.g., Facebook's mcrouter with LeaseGet/LeaseSet), the lease is a time-bound token held by the client; the server does not proactively cancel it when a delete arrives. If the lease holder performs LeaseSet after the delete has been processed but before the lease expires, the new value is accepted and the stale data reappears until TTL expiry or the next invalidation.
This means the staleness window can extend beyond the invalidation arrival time by up to the remaining lease duration. To bound this, some deployments combine short lease TTLs (e.g., 10–50 ms) with a pre-delete step: the invalidation publisher issues a delete, waits a brief quiesce period, then allows repopulation. Without that coordination, the only guarantee is that the value will eventually be invalidated again.