Speculative Retry Interaction With Read Repair Consistency Boundary

Short answer

Speculative retry and read repair are independent mechanisms, and neither one guarantees the other. Speculative retry fires an extra read at another replica when the coordinator's first request is slow; it does not trigger, await, or even know about read repair. Read repair chance (on versions where it still applies) only governs probabilistic background repair on CL ONE reads. The result: at CL ONE, a speculative retry can return a different value than the first responder, and whether divergence ever gets repaired is a separate, asynchronous question. At QUORUM and above, digest-mismatch resolution reconciles replicas before returning, so the retry becomes a latency optimization rather than a consistency variable.

How the interaction actually plays out

Three facts explain the non-determinism you're seeing:

• Speculative retry is latency-triggered, not consistency-triggered. Policies like 99p or 10ms are computed from coordinator read latency percentiles. There is no defined correlation between the retry firing and any repair completing — the retry simply asks another replica and takes whichever answer arrives first.
• At CL ONE, replicas may be divergent. Background read repair (where read_repair_chance applies) is probabilistic and asynchronous. If one replica missed a write, the first responder might be the stale replica and the retried replica the fresh one — or vice versa. Repeated reads can flap between old and new values.
• Pending hinted handoffs widen the window. A node holding undelivered hints means some replica acknowledged writes it never received. Until hints are delivered (bounded by max_hint_window_in_ms and actual delivery), CL ONE reads routed to that replica are stale, and speculative retry randomly lands on either side of the divergence.

Consistency level behavior

At QUORUM / LOCAL_QUORUM, Cassandra compares digests across the required replicas; on mismatch it fetches full data and reconciles before returning. Speculative retry at these levels cannot return stale data — it can only add load and occasionally tail latency. This is why the standard recommendation is QUORUM reads paired with QUORUM writes: the retry policy stops influencing correctness entirely.

One caution on the ALWAYS retry policy: it can mask the symptom (you more often get the fresh answer) while doubling read load, without fixing the underlying divergence.

Version sensitivity

This behavior is version-dependent. In Cassandra 3.x, read_repair_chance was deprecated in favor of blocking read repair driven by consistency level, and 4.x further changed repair and monotonic-read behavior. Advice tuned for 2.x semantics may not apply to your cluster — confirm your version before changing table options.

Verifying on your cluster

Check the actual table settings:

DESCRIBE TABLE myks.mytable;  -- speculative_retry and read_repair settings

To observe the interaction directly, diverge two replicas deliberately (e.g., write while one node is down), then run repeated CL ONE reads with TRACING ON in cqlsh and watch whether retried requests return differing results. Repeat at CL QUORUM on the same data — you should see the newest value returned deterministically via digest reconciliation. Finally, check nodetool netstats and hinted handoff metrics to confirm whether pending hints correlate with the stale reads you observed.

Practical recommendation

If you need deterministic reads, don't tune the retry/read-repair interaction — remove it from the correctness equation: use LOCAL_QUORUM reads and writes, keep speculative retry as a pure latency hedge, and monitor hint backlogs so divergent replicas heal quickly. Reserve CL ONE for data where transient staleness is acceptable.

One detail that would sharpen this: your Cassandra version and the table's current speculative_retry setting, since the read repair model differs enough across 2.x/3.x/4.x to change which knob matters.