Online Resharding in Vitess Using vreplication: Step‑by‑Step Guide

Online resharding with Vitess vreplication

You can move a keyspace from one shard layout to another without stopping traffic by creating a vreplication workflow that streams data from the source shards to the target shards, then cutting over once the copy is complete.

How vreplication works

Vitess runs a vreplication controller on each target tablet. It reads rows from the source tables using the vstream protocol, applies any filter, and writes them to the target tables. While the workflow is running, the original shards continue to serve reads and writes; the target shards catch up via replication lag. When lag reaches zero, a cutover swaps the serving type so that VTGate routes traffic to the new shards.

Worked example: resharding the commerce keyspace

1. Ensure you have access to a vtctld endpoint (default port 15999) and that your Vitess version is 21.0 or later.
2. Create the workflow. Run this command on a machine that can reach the vtctld server:

vtctldclient -server localhost:15999 CreateVReplicationWorkflow \
  -workflow commerce_reshard \
  -sourceKeyspace commerce \
  -sourceShard 0 \
  -targetKeyspace commerce \
  -targetShard -80 \
  -tables 'customer,order' \
  -filter ''

You need the vtctld ACL permission VReplicationAdmin to run this.
Start the workflow:

vtctldclient -server localhost:15999 StartVReplicationWorkflow -workflow commerce_reshard

Monitor progress. Check lag and state:

vtctldclient -server localhost:15999 ShowVReplicationWorkflow -workflow commerce_reshard

Look for State: Running and LagSeconds decreasing toward zero. You can also watch the _vt.vreplication table for errors.
When LagSeconds shows 0 for a sustained period (e.g., 30 seconds), execute the cutover:

vtctldclient -server localhost:15999 CutoverVReplicationWorkflow -workflow commerce_reshard

This switches the tablet type of the target shard to PRIMARY and the source shard to REPLICA, causing VTGate to route new traffic to the new shard.
After cutover, stop the workflow to free resources:

vtctldclient -server localhost:15999 StopVReplicationWorkflow -workflow commerce_reshard



Limits and common mistakes

Disk space. The target tablets must have enough free space for a full copy of the selected tables. Monitor df -h on the target host or the Vitess tablet metrics tablet_disk_used.
Network bandwidth. The initial copy phase transfers all rows; ensure the network between source and target tablets can sustain the throughput, otherwise lag will stall.
Primary key requirement. Every table involved must have a unique key or primary key; vreplication uses it to generate row‑based events. Tables without a unique key will cause the workflow to fail with an error about missing key.
Foreign keys. Vitess does not enforce foreign‑key constraints during vreplication; you must defer them or handle them manually after cutover.
Unchanged primary‑key type. You cannot alter a table’s primary‑key column type as part of the reshard; the workflow will reject the schema mismatch.
Omitting -filter. If you only need a subset of tables, still specify -tables but leave -filter empty (as shown) or provide a SQL‑like filter. Leaving the flag out causes vreplication to attempt to copy all tables in the keyspace, creating unnecessary load.
 Forgetting to update routing. After cutover, VTGate automatically routes to the new shard, but any application‑level connection pools that hard‑code shard names must be updated, otherwise writes will go to the old shard and diverge.
Cutting over too early. Running CutoverVReplicationWorkflow while lag is non‑zero can lose transactions that are still in flight on the source. Verify lag is zero and stable before cutover.
Leaving the workflow running. After a successful cutover, the vreplication workflow continues to replicate from source to target, wasting CPU and I/O. Always run StopVReplicationWorkflow once you have confirmed the new shard is serving traffic.

Practical verification steps

Run ShowVReplicationWorkflow and confirm State: Running and LagSeconds == 0.
Check the _vt.vreplication table: SELECT workflow_name, state, error_msg FROM _vt.vreplication WHERE workflow_name = 'commerce_reshard'; – error_msg should be NULL.
After cutover, query a row that only exists in the resharded tables (e.g., insert a test row before cutover and verify it appears on the new shard).
Monitor VTGate metrics: VTTabletType for the target shard should show PRIMARY and the source shard REPLICA.