Managing Database Growth with Vitess VReplication

The Scaling Wall: When a Single Shard Isn't Enough

Most database migrations involve a "maintenance window"—a period of read-only access or total downtime while data is exported and imported into a larger instance. For high-traffic applications, this window is a liability. When a single MySQL shard reaches its storage or CPU limit, you need to split that data across multiple new shards without stopping the world.

The core challenge is maintaining data consistency while moving millions of rows. If you stop writes, you lose availability. If you don't, the destination shard is outdated the moment the copy finishes. Vitess solves this using VReplication, a mechanism that treats data migration as a continuous stream rather than a one-time event.

How VReplication Handles Online Migration

VReplication enables "online" resharding by decoupling the initial data movement from the real-time updates. It operates in three distinct phases to ensure the destination is a mirror of the source before any traffic shifts.

1. The Copy Phase

Vitess takes a snapshot of the source shard. It reads the existing rows and writes them to the destination shards based on a new keyspace mapping. Because the application is still writing to the source, this snapshot is immediately stale, but that is expected.

2. The Catch-up Phase

To close the gap, Vitess leverages the MySQL binary log (binlog)—the record of every change made to the database. VReplication streams these binlogs from the source to the destination. The destination shard applies these mutations in order, effectively "playing back" everything that happened during the Copy phase until the lag between source and destination is near zero.

3. The Cutover

Once the destination is caught up, Vitess performs a Cutover. This is the only moment of restricted access. Vitess briefly blocks writes to the source, ensures the final few binlog events are applied to the destination, and updates the routing table in VTGate (the Vitess proxy). Traffic is then routed to the new shards, and the source is decommissioned.

Practical Example: Initiating a Reshard

Resharding is typically managed via the vtctldclient tool. This tool communicates with the cluster manager to orchestrate the VReplication workflow across multiple tablets.

To split a shard (e.g., shard -8) into two new shards (-32 and -33), a technician would run a sequence similar to this on a management node with appropriate administrative permissions:

# 1. Create the resharding workflow
vtctldclient Reshard -shard -8 -target-shards 2

# 2. Move the data (This triggers the Copy and Catch-up phases)
vtctldclient MoveTables -workflow <workflow_id>

# 3. Verify the VReplication status
vtctldclient GetWorkflowStatus -workflow <workflow_id>

Expected Check: In the status output, look for the VReplication state. It will transition from Copying to CatchingUp. Do not proceed to cutover until the lag is consistently low (typically measured in milliseconds).

Trade-offs and Operational Constraints

While VReplication removes the need for long outages, it introduces specific technical pressures:

• Binlog Dependency: VReplication requires binlog_format=ROW on the underlying MySQL instances. If binlogging is disabled or configured for statements only, the migration will fail.
• Write Amplification: During the Copy and Catch-up phases, the source shard must handle both production traffic and the read-load of the migration process. High write volume can increase binlog lag, extending the time it takes to reach the Cutover phase.
• The Sharding Key Risk: VReplication moves data, but it cannot fix a bad sharding key. If your key causes a "hot shard" (where one shard receives 90% of traffic), splitting that shard will only provide temporary relief before the new hot shard also saturates.

Verifying the Result

After the cutover, verify the migration by querying the vtgate proxy. Run a count query across the keyspace to ensure the total row count matches the pre-migration state. Because VTGate abstracts the routing, your application code should not need to change; it simply sees a more responsive database as the load is distributed across the new shards.