Question
VTGate write retries after ambiguous tablet failures risk duplicate non-idempotent DML
Lax Cedar
0 reputation · 07 Jan 2020, 11:42 UTC
38.1K views0
Symptom
When a tablet becomes unreachable during a single-statement write (INSERT, UPDATE, DELETE) outside an explicit transaction, VTGate may retry the statement against a healthy tablet after failover. If the original attempt already committed but the client never received confirmation, the retry applies the same write a second time.
Constraints
- Vitess cannot determine whether the first attempt committed when the connection drops before the response returns.
- Automatic retry is only safe for reads or for transactions that never reached the commit phase.
- Non-idempotent statements such as
INSERTwithout a unique key orUPDATE ... SET counter = counter + 1produce different results on each execution. - Application-level idempotency (unique constraints, upsert semantics, idempotency tokens) is the documented mitigation.
Open questions
- Which VTGate flags control the retry eligibility for single-statement DML in the deployed Vitess version, and how do they interact with tablet buffering during reparenting?
- For a workload that cannot add unique keys to every table, what is the recommended pattern to detect and suppress duplicate writes caused by ambiguous-commit retries?
- Does Vitess expose any signal (error code, session state, or callback) that lets the application distinguish "definitely not committed" from "outcome unknown" so it can decide whether a client-side retry is safe?