Live Resharding MySQL Shards with Vitess VReplication

The problem: a single shard becomes a bottleneck

As traffic grows, a Vitess keyspace that lives in a single shard (e.g., shard 0) can start to show latency spikes and limited write throughput. The usual fix—adding more shards—requires moving data without taking the application offline.

Thesis: VReplication lets you copy data to a new shard while keeping source and target in sync, enabling a cut‑over with zero downtime

Vitess VReplication is a built‑in mechanism that streams changes from a source tablet to a target tablet. By creating a VReplication workflow you can bootstrap the target with a snapshot, then continuously apply incremental changes. When the lag drops to zero you can switch traffic to the new shard and later remove the old one.

Section 1: How VReplication works

VReplication treats the source as a producer and the target as a consumer. The workflow proceeds in three phases:

1. Snapshot – the target copies the existing data from the source.
2. Copy – any changes that happened during the snapshot are applied.
3. Realtime – the workflow streams ongoing binlog events, keeping source and target identical.

All of this happens inside Vitess; you only need to issue a vtctl command to start the workflow.

Section 2: Preparing the environment

Before you begin, verify that both the source and target shards have enough free disk space to hold the full dataset during the snapshot phase. A quick check is:

# Run on each MySQL host (requires sudo or mysql admin)
mysql -e "SHOW VARIABLES LIKE 'datadir';"
# Then check free space on that directory.

You also need a running Vitess cluster (the official docker‑compose demo works fine) and the vtctl client available in your PATH.

Section 3: Starting the VReplication workflow

The worked example below moves the commerce keyspace from shard 0 to shard 1. Adjust the placeholders for your environment.

# Run on any host with vtctl access (needs vitess admin privileges)
vtctl \
  -server localhost:15999 \
  VReplicationCreate \
    -keyspace commerce \
    -source_shard 0 \
    -target_shard 1 \
    -workflow_name commerce_reshard \
    -tablet_types "primary,replica" \
    -exclude_tables "" \
    -include_tables ""

What this does:

• Creates a workflow named commerce_reshard.
• Starts a snapshot of commerce from shard 0 to shard 1.
• After the snapshot, switches to realtime replication.

You can watch the progress with:

vtctl -server localhost:15999 VReplicationShowWorkflow -workflow commerce_reshard

The output shows the current phase (Snapshot, Copy, Realtime) and any errors.

Section 4: Monitoring lag and performing the cut‑over

When the workflow reaches the Realtime phase, check replication lag:

# Vitess debug endpoint (run on any vtgate host or via curl)
curl http://localhost:15000/debug/vars | grep -A2 VReplication

Look for a field like VReplication_commerce_commerce_reshard_lag. When the value is consistently 0 (or a few milliseconds), the target is caught up.

At that point you can switch traffic:

vtctl -server localhost:15999 SetShardIsPrimary commerce/1
# Then update your application’s routing or Vitess cell alias to point to shard 1.

After confirming the application works correctly, you can stop the workflow and remove the old shard:

vtctl -server localhost:15999 VReplicationStop -workflow commerce_reshard
vtctl -server localhost:15999 DeleteShard -keyspace commerce -shard 0

Trade‑offs and limitations

During the snapshot and copy phases the source and target both store the full dataset, temporarily doubling disk usage. Network traffic increases because VReplication streams binlog events continuously. Monitor both metrics to avoid saturating your infrastructure.

Excessive lag can delay the cut‑over and increase the risk of applying stale data if you switch too early. Always verify that lag is near zero before changing traffic.

Actionable closing

If you are facing a hot shard, try the workflow above on a staging cluster first. Use the VReplicationShowWorkflow command to confirm each phase, check the /debug/vars endpoint for lag, and only then promote the new shard. Once traffic is switched, the old shard can be safely retired, freeing resources for future growth.