Online Resharding in Vitess: Zero‑Downtime Shard Splitting

Problem: Rebalancing Shards Without Downtime

When a Vitess cluster runs out of capacity, you must split a keyspace into more tablets. Traditional approaches require stopping writes, copying data, and restarting the application, which is unacceptable for production services. Vitess provides a built‑in, zero‑downtime mechanism called online resharding that leverages VSchema and VReplication.

How Vitess Sharding Works

Vitess shards a keyspace into tablets, which are individual MySQL instances. VTGate, the query router, receives a SQL statement and uses the VSchema to map the target table to a specific tablet based on the defined keyspace id range. The VSchema stores the mapping in the cluster’s metadata, and VTGate routes queries accordingly. Application code does not need to know the shard layout; it simply connects to VTGate.

Online Resharding Mechanics

The resharding workflow consists of three phases:

1. Schema update – a new VSchema entry defines the source shard and the target shard for the table. This change is applied cluster‑wide with `vtctl UpdateVSchema`.
2. Replication start – `vtctl StartReplication` creates a VReplication job that streams rows from the source tablet to the newly created target tablet. The job tracks replication positions and reports lag.
3. Catch‑up and cutover – once the target tablet’s position catches up, VTGate is instructed to route writes to both source and target (dual‑write) and finally to the target only, completing the move atomically.

All commands are executed on any cluster member that has the appropriate admin privileges; VTGate automatically propagates the changes.

Worked Example: Splitting the customers Keyspace

Suppose the `customers` keyspace currently uses two tablets covering key ranges `0‑5000000` and `5000001‑∞`. The goal is to create four shards.

# 1. Create new tablets (run on each MySQL instance)
vtctl CreateTablet --keyspace=customers --table=customers --shard=customers_0_2500000
vtctl CreateTablet --keyspace=customers --table=customers --shard=customers_2500001_5000000
vtctl CreateTablet --keyspace=customers --table=customers --shard=customers_5000001_7500000
vtctl CreateTablet --keyspace=customers --table=customers --shard=customers_7500001_∞

These commands require the `vtadmin` role and must be executed on a node that can reach the MySQL instances.

# 2. Update VSchema to add the new shard mappings

1. Define the new source‑target pairs in a JSON file, e.g. `customers_0_2500000` → `customers_0_2500000` is not needed; instead we map the old shard `0‑5000000` to both old and new shards `0‑2500000` and `2500001‑5000000`.
2. Apply the update:

vtctl UpdateVSchema --keyspace=customers --table=customers \
  --definition='{
    "sources": [
      {"shard": "0-5000000", "tablet": "customers_0_5000000"}
    ],
    "targets": [
      {"shard": "0-2500000", "tablet": "customers_0_2500000"},
      {"shard": "2500001-5000000", "tablet": "customers_2500001_5000000"}
    ]
  }'

# 3. Launch replication from the old tablet to each new tablet:

1. Start replication for the first half:

vtctl StartReplication --keyspace=customers --table=customers \
  --source-shard=0-5000000 --target-shard=0-2500000

# 4. Verify replication lag; it should stay below a few seconds in a healthy cluster. Use:

vtctl GetReplicationPosition --keyspace=customers --table=customers --shard=0-2500000

If lag exceeds the configured threshold, pause or investigate before proceeding.

# 5. After both halves catch up, switch routing:

1. Update VSchema to route writes to both old and new shards (dual‑write).
2. Perform a short cut‑over window where VTGate routes all writes to the new tablets only.
3. Optionally, drop the old tablets after confirming no traffic.

This sequence preserves transactional consistency within each shard and avoids application downtime.

Trade‑offs and Limitations

Online resharding improves availability but introduces operational concerns:

• Write amplification: each row is written twice during the dual‑write phase, increasing load on the source tablets and potentially affecting throughput.
• Replication lag risk: if the target tablets cannot keep up, VTGate may fall behind, causing stale reads. Continuous monitoring of `VReplication` position is essential.
• Cross‑shard transactions: Vitess only guarantees atomicity within a single shard. Distributed transactions require two‑phase commit coordination, which can be slower and more error‑prone.
• Schema validation: an incorrect VSchema definition can misroute queries, leading to data loss or inconsistent results. Always validate the keyspace‑id ranges against existing tablet exports.

Actionable Closing Checklist

Before running resharding in production, follow this checklist:

1. Staging test – replicate production data volume in a non‑production cluster and run the full resharding flow. Confirm that replication lag stays within acceptable bounds (e.g., < 5 seconds).
2. Permissions check – ensure the executing user has the `vtadmin` role on VTGate and can access all MySQL instances.
3. VSchema validation – run `vtctl VerifyVSchema` (or equivalent) to ensure the new definitions match the actual key ranges reported by `vtctl GetTabletExports`.
4. Monitoring setup – enable metrics for `VReplication` lag, CPU usage on source tablets, and VTGate query latency. Set alerts for lag > 10 seconds.
5. Rollback plan – if the cut‑over fails, revert the VSchema to the original mapping and stop the replication jobs. The old tablets remain available, so no data loss occurs.

By adhering to these steps, you can rebalance your Vitess cluster with minimal impact on users and maintain data consistency throughout the operation.