Stale cache results during read repair with consistency level operations

Direct Answer

Read repair does not guarantee that a stale row cache entry is cleared on every node, including the coordinator, as part of a single repair operation. The repair mutation is sent to the out‑of‑date replicas and they invalidate the affected row in their own row cache. The coordinator’s cache is not touched by that repair; it will only be refreshed on the next write that passes through it or when the cached entry expires or is evicted. Therefore, after a read repair a subsequent read at the same consistency level can still return a stale value if the coordinator’s row cache still holds the old partition.

Changing row_cache_size_in_mb to zero removes the possibility of a stale coordinator cache, but does not affect the replicas’ caches. The key_cache_size_in_mb setting only caches partition‑index locations; it does not hold cell data and therefore does not create a stale‑data window.

No documented combination of consistency level and compaction strategy (e.g., LCS vs. SCS) guarantees the elimination of stale cache results after a read repair. The observed behavior is largely version‑dependent, with differences noted between Cassandra 3.x and 4.x in how cache invalidation is performed and how long a stale entry may persist.

Likely Explanation

In practice, a read at QUORUM or LOCAL_QUORUM that triggers read repair will contact all replicas in the read quorum. The coordinator forwards the newer value to any replica that is lagging and instructs it to apply the mutation, which includes invalidating its row cache entry. The coordinator, however, keeps its own cache entry until a new write invalidates it or the entry ages out. Because the coordinator may serve a cached row before contacting the repaired replica, a client can observe the old value immediately after a repair.

Compaction strategy influences how quickly on‑disk inconsistencies are merged, but it does not affect the in‑memory row cache. Even with LCS, a stale cache entry can survive until eviction or time‑based expiry.

Steps to Verify in Your Cluster

1. Check the value of row_cache_size_in_mb in cassandra.yaml. If it is greater than zero, a stale coordinator cache window is possible.
2. Enable tracing for a test read: TRACING ON in cqlsh, perform a write that creates divergence (e.g., INSERT at ONE), then read at QUORUM to trigger repair, and finally read again at QUORUM.
3. Inspect the trace output for “row cache hit” entries versus “read repair” activity to see whether the coordinator served a cached row.
4. If you want deterministic post‑repair reads, set row_cache_size_in_mb to 0 and restart the node(s); repeat the test.

Missing Diagnostic Detail

To give a more precise recommendation, I need to know the exact Cassandra minor version you are running (e.g., 3.11.x, 4.0.x, 4.1.x). Version‑specific code changes can alter the timing of cache invalidation and the interaction with read repair.