Vitess Online Resharding With VSchema and VReplication

Useful answer first

Resharding in Vitess is VSchema-driven. You declare a new sharding layout for the keyspace, start a Reshard workflow that builds VReplication streams from source shards to target shards, let SplitClone do the initial copy and VReplication catch up ongoing changes, then complete the workflow to switch VTGate routing to the new shards. The operation is online, but it requires a stable sharding key, monitoring of replication lag, and application queries that stay shard-local.

How the mechanism works

Vitess routes queries through VTGate using the VSchema. The VSchema defines the keyspace, its sharding column, and the mapping of key ranges to physical shards. VReplication is Vitess' built-in change data capture that replicates rows between MySQL-compatible tablets using binlog positions. SplitClone is the initial bulk copy used by resharding to seed target shards.

A typical reshard moves from a 2-shard layout to a 4-shard layout. The sharding column must be deterministic and present in every row. The workflow creates target shards, copies data, replicates changes, and finally updates routing.

Worked configuration example

Assume keyspace customer currently uses customer_id with shards -80 and 80-. You want four shards: -40, 40-80, 80-c0, c0-.

First inspect the current VSchema on a vtctl node with admin access:

vtctlclient -server <vtctld_addr> GetVSchema customer

Expected check: confirm sharded: true, split_column: customer_id, and the existing shard names.

Create a new VSchema that declares the target layout. The reshard workflow uses a temporary keyspace name to hold the new shards. The VSchema JSON is applied via vtctlclient:

vtctlclient -server <vtctld_addr> ApplyVSchema -vschema_file customer_reshard.vschema.json

An illustrative customer_reshard.vschema.json fragment:

{sharded: true, split_column: customer_id, split_column_type: UINT64, keyspace_id_type: UINT64, tables: {customer: {column_vindexes: [{column: customer_id, name: customer_id}]}}}

Start the reshard workflow from a vtctl client with privileges to create VReplication streams and modify VSchema:

vtctlclient -server <vtctld_addr> Reshard -keyspace customer -workflow customer_reshard -source_shards -80,80- -target_shards -40,40-80,80-c0,c0- -cells <cell1,cell2>

Risks: this creates VReplication streams on each source tablet. Initial SplitClone consumes read I/O and network. Write amplification occurs because writes are applied to source shards and then replicated to targets.

The workflow proceeds in phases:

• SplitClone copies existing rows from each source shard to the appropriate target shards based on the new key ranges.
• VReplication streams are started and track binlog positions to apply ongoing writes.
• Status is monitored until all streams report caught up.

Check progress:

vtctlclient -server <vtctld_addr> Reshard -action status -keyspace customer -workflow customer_reshard

Expected check: each source-to-target stream shows lag near zero and state Running. Also inspect:

vtctlclient -server <vtctld_addr> VReplicationStatuses customer

When caught up, complete the workflow:

vtctlclient -server <vtctld_addr> Reshard -action complete -keyspace customer -workflow customer_reshard

Completion updates the VSchema so VTGate routes reads and writes to the new shards and stops the VReplication streams. Verify routing with vtgate explain for a sample query using a known customer_id.

Limits and practical constraints

Resharding requires a deterministic sharding key with good distribution. A high-cardinality but non-uniform key creates hot shards after the split. The operation assumes MySQL-compatible tablets, healthy VReplication monitoring, and sufficient disk and network capacity for the initial copy and dual writes.

Temporary storage overhead exists while both source and target shards hold the data. Write amplification increases latency during the catch-up phase. Foreign keys and cross-shard transactions are not supported by Vitess; resharding does not fix application logic that assumes single-shard queries. Cross-shard joins remain unsupported.

Behavior is version sensitive. Reshard workflow flags and VSchema syntax have evolved across Vitess releases, so validate commands against the deployed version.

Common mistakes

Choosing a sharding key that is not uniformly distributed leads to hot shards and uneven load after cut-over. Forgetting to update application-level routing rules or VSchema for services that reference the keyspace can cause queries to target stale shard names.

Running resharding while long-running transactions are active can cause replication lag and cut-over failures. Avoid large bulk loads during the catch-up window.

Skipping verification of VSchema before and after the change risks routing to wrong shards. Always inspect VSchema with GetVSchema and confirm VReplication lag is zero before completing.

Verification checklist

• Inspect VSchema before and after: vtctlclient GetVSchema keyspace confirms sharding column and shard names.
• Monitor VReplication status until all streams are caught up and no errors are reported.
• Verify VTGate routing for representative keys after completion using vtgate explain.