mutations_sync = 1 vs Asynchronous Mutations for Schema Consistency

Answer to the core questions

1. Does mutations_sync = 1 guarantee that all replicas have physically applied the change before the command returns?

• No. The setting only blocks the client session until the replica that executed the ALTER reports that its own mutation is finished. The mutation is still applied asynchronously on the other replicas.

2. What are the performance implications for insert throughput when synchronous mutations are enforced on high‑volume tables?

• Waiting for a merge to finish on a replica ties up that replica’s merge thread and I/O for the duration of the mutation, which can increase write latency and reduce effective insert throughput.
• Large mutations can block client connections and may trigger timeouts, especially under heavy write load.

Confirmed behavior

• With mutations_sync = 0 (default), the ALTER command returns immediately after registering the mutation in ZooKeeper. Physical rewriting occurs in the background.
• With mutations_sync = 1, the client session blocks until the executing replica reports is_done = 1 for that mutation. No cluster‑wide barrier is enforced.

Implications for consistency

Because the mutation is applied per‑replica, a query that hits a different replica immediately after an ALTER with mutations_sync = 1 can still read the pre‑mutation data. Full consistency across the cluster requires waiting for all replicas to finish the mutation, which can be verified via system.mutations on each replica.

Impact on insert throughput

When the merge thread is busy finishing a mutation, the replica is temporarily less available for new inserts. The longer the mutation, the greater the contention. In high‑volume workloads, this can noticeably lower the number of rows inserted per second and increase write latency.

Practical steps for your scenario

1. Keep mutations asynchronous (default) unless you need read‑after‑write guarantees on the same replica.
2. After issuing an ALTER, query system.mutations on each replica to confirm is_done = 1 before performing a read that must see the mutation.
3. For read‑after‑write consistency, route the SELECT to the same replica that executed the ALTER or wait until system.mutations shows all replicas finished.
4. Monitor merge_mutations and insert_rows metrics to assess the throughput impact when toggling mutations_sync.

Diagnostic detail needed

To tailor the recommendation further, could you let us know the typical size of the mutations you run (e.g., number of rows or part size) and the number of replicas per table?