Using Memcached CAS to Safely Update Counters Under Concurrency

The problem: lost increments when multiple clients update a counter

When two or more application instances read a numeric value from Memcached, increment it locally, and write it back with a plain set, both may read the same old value before either writes. The second write overwrites the first, so one increment is lost.

How Memcached’s CAS operation helps

Memcached stores a 64‑bit “check‑and‑set” token with each item. The gets command returns the value together with its current token. A subsequent cas command succeeds only if the token supplied by the client matches the token still stored on the server. If another client has changed the item in the meantime, the token differs and the server replies EXISTS, signalling a conflict.

Worked example (pseudo‑code)

// client library that exposes gets and cas
function incrementCounter(key) {
  while (true) {
    const result = memcached.gets(key);   // {value, casToken}
    const newValue = result.value + 1;
    const status = memcached.cas(key, newValue, result.casToken);
    if (status === 'STORED') {
      return newValue;   // success
    }
    // token mismatch → retry
    // optional: add a small back‑off here
  }
}

The loop retries until the cas call returns STORED. Under contention some iterations may fail, but the counter is guaranteed to reflect every increment that successfully passes the token check.

Trade‑offs and limits of CAS

• Each attempt needs two round‑trips (gets + cas), adding latency.
• Under high contention clients may spin many times, consuming CPU and increasing tail latency.
• Memcached does not guarantee fairness or bound the number of retries; a pathological workload could cause starvation.
• CAS only protects a single key; atomic updates across multiple keys still require external coordination.

Practical way to verify the behavior locally

1. Start a Memcached instance on the default port: memcached -p 11211 (run in a terminal).
2. Open a telnet session: telnet localhost 11211.
3. Store a counter: set counter 0 0 1\r\n0\r\n (flags=0, exptime=0, 1 byte, value “0”).
4. Retrieve the token: gets counter\r\n. You should see something like VALUE counter 0 1 12345\r\n0\r\nEND\r\n where “12345” is the CAS token.
5. In a second telnet session, change the value: set counter 0 0 1\r\n1\r\n.
6. Return to the first session and try to CAS with the old token: cas counter 0 0 1 12345\r\n2\r\n. The server replies EXISTS, indicating the token mismatch.
7. Repeat the gets to fetch the new token, then issue a matching cas; you will receive STORED.

These steps show the token‑based check in action without writing any code.

When to reach for something else

If you expect heavy contention on a counter, or you need to update several keys atomically, consider:

• Using a dedicated atomic counter service (e.g., Redis INCR).
• Queuing increments and applying them with a single writer.
• Introducing a lightweight distributed lock (e.g., based on SET NX with expiry) around the critical section.

For low‑to‑moderate contention, the CAS retry loop remains a simple, dependency‑free way to get correct increments straight from Memcached.

Actionable takeaway

Add a gets/cas retry loop around any single‑key mutable state you store in Memcached. Monitor the cas_misses metric (exposed via stats) to spot excessive retries, and introduce a small exponential back‑off if the miss rate climbs. This gives you correct concurrent updates without moving to a heavier data store, while keeping the operational simplicity of Memcached.