Vitess Online Resharding with VReplication: A Staged Cut-Over Guide

The problem: moving a shard while writes keep arriving

Resharding a live Vitess keyspace is an operational decision, not a button press. The central risk is that VSchema routing changes are cluster-wide, and VReplication lag can spike under write bursts. If you flip routing before the target is caught up, vtgate will send queries to a shard that is missing rows. A safe migration therefore requires staged verification gates before traffic moves.

This guide walks through the engineering decision of using VReplication to move data and traffic from a source shard to a new target shard or keyspace with minimal downtime. It assumes a MySQL-compatible Vitess deployment where applications connect through vtgate.

Desired outcome and scope

The goal is to migrate data and live traffic from a source keyspace shard to a target shard or keyspace while preserving consistency. Reads and writes should continue through vtgate during the move, accepting only a brief read-only window during the final switch. The scope covers provisioning, replication catch-up, routing cut-over, and decommissioning the source.

Key terms used below:

• Keyspace: a logical database in Vitess.
• Shard: a physical partition of a keyspace, typically one or more MySQL instances.
• VSchema: the Vitess schema that defines how tables map to keyspaces and shards.
• VReplication: the Vitess workflow that streams row changes from a source to a target.
• vtgate: the query router that applications connect to.
• vttablet: the tablet process that manages each MySQL instance.
• vtctld: the control plane for topology and workflows.

Prerequisites before you touch routing

Do not start the procedure until all of the following are in place:

1. A running Vitess cluster with vtgate, vttablet, vtctld, and a topology service.
2. A defined VSchema for both the source and target keyspaces, including sharding keys and routing rules.
3. VReplication enabled on the target tablets.
4. Sufficient capacity headroom for dual writes and replication traffic, because disk I/O and latency increase during migration.
5. Recent backups of the source shards.
6. Monitoring for replication lag and tablet health.
7. An application that is tolerant of a brief read-only window and that uses idempotent writes, since cut-over is not automatically atomic.

Also confirm that no schema changes are running against the source during the migration window.

Focused procedure

1. Provision the target

Create the target keyspace and shards, deploy the required vttablets, and verify that every target tablet reaches a serving state. Until the tablets are healthy, replication streams cannot reliably apply changes.

2. Define target routing

Update the VSchema on the target keyspace so it defines the same table routing and sharding keys as the source. This prepares vtgate to resolve queries correctly once traffic switches. Do not change global routing rules yet.

3. Start VReplication streams

Create VReplication streams from the source to the target for each shard. These streams copy existing rows and then continuously apply new changes. The exact command syntax and flags differ across major Vitess releases, so confirm the workflow subcommand and arguments against your deployed version before execution.

Example status check on the vtctld host (or any host with vtctldclient configured), using administrative credentials:

vtctldclient --server <vtctld-host>:<port> workflow --keyspace <source-keyspace> status <workflow-name>

Expected check: the reported position lag should trend toward zero and the stream state should indicate active replication. If lag grows under load, wait for a sustained low-lag window rather than proceeding.

4. Validate consistency before cut-over

Compare row counts and checksums on representative tables between the source and target shards. This is the last chance to catch copy errors before routing changes. If checksums diverge, stop the stream, investigate, and re-copy the affected tables.

5. Cut over writes

When replication is caught up and consistency checks pass, update the VSchema to route writes to the target. Because this change is cluster-wide, perform it when the application can tolerate a brief read-only window. After the change, vtgate will direct new writes to the target shard.

6. Verify traffic and decommission

Exercise read and write traffic through vtgate and confirm that queries are served by the new shard. Monitor error rates and latency. Only after sustained healthy operation should you stop VReplication streams and decommission the source tablets.

Expected checks

Phase	Check	Expected result
Pre-cut-over	VReplication position lag	Near zero and stable
Pre-cut-over	Row counts and checksums	Match between source and target on sampled tables
Pre-cut-over	Target tablet serving state	All target tablets healthy and serving
Post-cut-over	vtgate error rate and latency	No sustained increase beyond baseline
Post-cut-over	Query routing	Application traffic resolves to the new shard

Recovery options

If something goes wrong, the recovery path depends on whether writes have already landed on the target.

Situation	Action	Risk
Lag growing before cut-over	Pause or stop VReplication streams; revert VSchema to route back to source	Low, because writes are still on source
Checksum divergence before cut-over	Stop the stream, investigate, and re-copy affected tables	Extended dual-write window
Errors immediately after cut-over	Revert VSchema to source if no writes have landed on target	May require data reconciliation if writes already landed
Target data corruption after cut-over	Restore target tablets from backup	Downtime while restoring and re-replicating

Test the rollback path in a non-production environment before attempting it in production. Reverting VSchema after writes have been applied to the target is not a simple undo; it can require manual reconciliation.

Limitations and version sensitivity

• Cut-over is not atomic. If the application is not idempotent, a brief switch can produce duplicate writes.
• Resource usage doubles during migration because replication and dual serving increase disk I/O and latency.
• VSchema syntax and VReplication controls differ across major Vitess releases. Command names, flag semantics, and workflow subcommands should be confirmed against the deployed version before execution.
• No concurrent schema changes. DDL or online schema changes on the source during cut-over can cause replication errors.

Practical way to check the result

After cut-over, run a known read and write transaction through vtgate and verify that it executes against the target shard. Then review tablet serving graphs and topology state to confirm all target tablets are healthy. Finally, monitor vtgate query latency and error rates for a sustained period before decommissioning the source.

Because Vitess workflow commands and VSchema formats evolve, treat the command examples above as patterns to verify against your version's documentation rather than copy-paste invocations.