Vitess Resharding Guide: Diagnose, Verify, and Cutover Safely

Recognizable Condition

When a MySQL cluster running Vitess needs to rebalance shard capacity—by splitting a large shard, merging two shards, or moving a shard to a new tablet—you must ensure the operation completes without disrupting service.

Diagnostic Overview

Symptom	Likely Cause
High replication lag on source tablets during copy phase	Insufficient write capacity or heavy read traffic on source shards.
VTGate returns "shard not found" errors after cutover	VSchema key range mismatch or stale SrvVSchema entries.
Copy stalls on large rows or missing sharding key indexes	Very large rows or absent primary key on sharding column causing slow copy.
Source tablets report "tablet not healthy" alerts	Replication load exceeds tablet resources, often from throttling or high traffic.

Ordered Checks

1. Confirm resharding definition.

   vtctldclient ApplyResharding --topology-server=localhost:3306 \
         --create-shard=SOURCE_SHARD:START_KEY,END_KEY \
         --create-shard=TARGET_SHARD:START_KEY,END_KEY \
         --table=TABLE_NAME

   Run on a host with admin privileges on the Vitess topology server. Verify that SOURCE_SHARD and TARGET_SHARD cover disjoint key ranges.

2. Check current resharding status and lag.

   vtctldclient ReshardingStatus --topology-server=localhost:3306

   Look for replication_lag per tablet. All lag must be < 1 second before proceeding.

3. Validate source tablet health.

   vtctldclient TabletHealth --topology-server=localhost:3306

   Any tablet marked unhealthy must be recovered (e.g., restart tablet, fix replication errors) before cutover.

4. Inspect VSchema entries.

   vtctldclient SrvVSchema --topology-server=localhost:3306

   Confirm that the sharding key column for TABLE_NAME matches the column used in the resharding definition.

5. Run pre‑cutover validation.
   • Row count comparison: vtctldclient Query --shard=SOURCE_SHARD --sql="SELECT COUNT(*) FROM TABLE_NAME"
   • Checksum comparison (if supported): vtctldclient Query --shard=SOURCE_SHARD --sql="SELECT MD5(GROUP_CONCAT(CONCAT(col1, '|', col2))) FROM TABLE_NAME"

   Both source and target row counts (and checksums) should match before cutover.

Fixes Tied to Findings

1. High replication lag.
   • Throttle vreplication: vtctldclient SetReplicationThrottle --table=TABLE_NAME --rate=500k.
   • Run during low‑traffic windows or add more source write capacity.
2. Shard not found after cutover.
   • Refresh VTGate routing: vtctldclient ReshardingCutover --topology-server=localhost:3306.
   • Ensure all clients use connection pooling with retry logic to handle transient routing errors.
3. Copy stalls on large rows.
   • Break the table into smaller key ranges and run separate ApplyResharding operations.
   • Add or verify a primary key on the sharding column to avoid full table scans.
4. Unhealthy tablets.
   • Investigate tablet logs for replication errors.
   • Restart the tablet or re‑initialize its replication streams.

Escalation Criteria

• Replication lag remains > 5 seconds after throttling for more than 10 minutes → consider pausing the operation and scaling out source write capacity.
• Any tablet stays unhealthy for > 5 minutes after health checks → roll back to previous topology using vtctldclient ReshardingAbort and investigate root cause.
• Post‑cutover queries return errors from multiple shards → re‑inspect SrvVSchema entries; if mismatched key ranges are found, re‑apply the resharding definition and repeat cutover.
• Row‑count or checksum mismatches after validation → abort cutover, re‑run the copy phase, and verify data integrity before proceeding.

Practical Verification

1. After ReshardingCutover, run vtctldclient ReshardingStatus again. Lag must be zero and all tablets healthy.
2. Execute a targeted VTGate query filtered by the new shard key range and confirm the response originates from a tablet belonging to TARGET_SHARD (check tablet hostname or ID in query logs).
3. Perform a final row‑count or checksum comparison between source and target shards. If they differ, restore the source shard tablets from backup and investigate.
4. Inspect the topology service: vtctldclient SrvVSchema --topology-server=localhost:3306 and verify that the sharding_key_range for TABLE_NAME now points to TARGET_SHARD.

Limitations & Practical Tips

• Resharding adds read and write load to source tablets; throttle or schedule during off‑peak hours to avoid performance degradation.
• Cutover requires a brief routing switch; use Vitess connection pooling with automatic retries to hide transient errors.
• VSchema key errors are not corrected by resharding; ensure correct sharding column and index before starting.
• Very large rows (> 10 MB) or missing sharding key indexes can stall the copy phase and may need manual intervention.

By following the diagnostic steps, applying the fixes aligned with observed symptoms, and verifying each stage, you can safely rebalance Vitess shards while keeping your MySQL service available.