Vitess Shard Failover Guide: Promoting Replicas

Learn how to use vtctl FailoverShard to promote a replica to master in Vitess, ensuring minimal downtime and automatic vtgate routing updates.

When a MySQL master node in a Vitess cluster fails, write operations for that specific shard stop immediately. Manually reconfiguring replicas and updating application connection strings is error-prone and causes significant downtime. Vitess solves this through a managed failover mechanism that promotes a synchronized replica to master status and automatically updates vtgate routing without requiring application-side changes.

This guide explains how to execute a controlled shard failover using vtctl to ensure data integrity and minimize disruption during hardware failures or planned maintenance.

Prerequisites for Safe Failover

Before initiating a failover, verify the following conditions to prevent data loss or routing errors:

Healthy Replica: At least one replica must be in a Ready state.
Low Replication Lag: The target replica should have near-zero lag. Promoting a lagging replica will result in the loss of transactions that had not yet reached that replica.
Replication Filters: Ensure no table-level replication filters are active that would exclude critical tables from the replica; otherwise, the new master will be missing data.
Administrative Access: You must have the necessary permissions to run vtctl commands against the vtctld service.

Executing the Shard Failover

The failover process is managed by the FailoverShard command. This operation automates the promotion of a replica, updates the topology metadata, and reconfigures the routing layer.

Step 1: Identify the Optimal Candidate

Check the current status of the shard to determine which replica is the most up-to-date and healthy.

# Run from a management node with vtctl installed
# Replace {keyspace} and {shard} with your specific identifiers
vtctl status -keyspace user_app -shard c01

Step 2: Trigger the Promotion

Promote the chosen replica using the FailoverShard command. Vitess employs a "failover window" (defaulting to 30 seconds) to prevent "flapping," where the system repeatedly switches masters due to transient network glitches.

# Syntax: vtctl FailoverShard -keyspace {keyspace} -shard {shard} -replica {replica_name}
# Example: Promoting replica-1 for shard c01 in the user_app keyspace
vtctl FailoverShard -keyspace user_app -shard c01 -replica replica-1

Step 3: Metadata Propagation

Once executed, vtctld updates the topology store. The vtgate instances poll this store and redirect traffic to the new master. This propagation typically occurs within a few seconds.

Verifying the Failover Result

After the command completes, verify that the new master is accepting writes and that routing is accurate.

Confirm Role Change: Run vtctl status again. The promoted replica should now be listed as the master.
Verify Routing Layer: Connect via vtgate and query the shard metadata to ensure the routing IP matches the new master:
```
# Execute via SQL client connected to vtgate
SELECT * FROM information_schema.shards WHERE shard = 'c01';
```
Test Write Consistency: Perform a dummy write through vtgate and read it back to confirm the pipeline is functional:
```
INSERT INTO test_table (id, val) VALUES (1, 'failover_test');
SELECT * FROM test_table WHERE id = 1;
```
Audit Logs: Inspect vtgate logs for a FailoverShard entry to ensure no underlying errors occurred during the transition.

Recovery and State Management

Failover changes the state of the cluster. Depending on the cause of the failure, you may need the following recovery steps:

Handling the Old Master

When the old master returns to the network, it will be in an inconsistent state relative to the new master. To restore high availability, you must reconfigure the old master as a replica of the current master. This is typically handled via vtctl RevertMaster or by rebuilding the node from a snapshot.

Rollback Considerations

If the failover was triggered accidentally or the new master is found to be corrupted, you cannot simply "undo" the operation. You must perform another FailoverShard to a different healthy replica or restore the shard from a backup snapshot. Because failover involves promoting a replica to a write-capable state, any writes made to the new master during the window will be lost if you revert to an older snapshot.

While the FailoverShard command triggers the promotion, it is important to clarify the role of the topology service (such as etcd or ZooKeeper) in this process. vtctld does not push updates directly to every vtgate instance; instead, it updates the central topology store.

vtgate instances rely on a polling mechanism or watches on the topology service to detect these changes. Depending on the configuration of the topology_service and the vtgate refresh interval, there can be a slight propagation delay. To minimize this window during a critical failover, verify that your topology service is highly available and that vtgate instances have low-latency network access to the store to ensure routing updates occur as close to real-time as possible.

Executing Vitess Shard Failover for High Availability