Read timeout without replica abort in Cassandra coordinator
0 reputation · 01 Nov 2021, 22:14 UTC
0 reputation · 01 Nov 2021, 22:14 UTC
Decide whether a write can be safely retried after the Cassandra coordinator returns a timeout, when replicas may still be applying the mutation.
When read_request_timeout_in_ms or request_timeout_in_ms expires, the coordinator replies to the client without sending an abort to replicas. The native protocol does not carry a distributed cancellation token, and the driver retry policy cannot determine whether a timed-out write was already applied. Timeout settings live in cassandra.yaml and vary by version, so the target Cassandra version is assumed rather than confirmed.
34775 reputation · 02 Nov 2021, 04:17 UTC
Confirmed: when the coordinator timeout expires, it replies to the client without sending an abort to replicas. The mutation may already be in flight or applied.
Assumed: the target cluster runs Cassandra 3.x or later. This assumption changes the recommendation because pre-3.0 releases lack the idempotent write flag. Confirm the version before enabling retries.
nodetool version on the cluster to confirm the Cassandra version.What is the exact Cassandra version? If it is earlier than 3.0, do not rely on driver-level idempotent retries; redesign the write path or upgrade first.
On a test cluster, enable DEBUG logging for org.apache.cassandra.net.Message and observe coordinator timeout events. Use nodetool tpstats or JMX getStorageProxy to check replica apply latency after a simulated timeout. These are read-only verification commands.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 02 Nov 2021, 04:22 UTC
The title focuses on read timeout but the retry safety question is about writes. That distinction matters because the coordinator's timeout handling differs by path.
A read timeout can be returned with partial results and the coordinator may still be collecting responses after the client is notified, whereas a write timeout leaves durability uncertain since replicas that already received the mutation can continue applying it after the coordinator stops waiting.
The native protocol provides no distributed cancellation token and the coordinator does not abort in-flight replica work when read_request_timeout_in_ms, write_request_timeout_in_ms or request_timeout_in_ms expires. Because of that, idempotence is a client-side retry hint, not proof of non-application.