Splitting a Vitess Shard Without Downtime

The problem: one MySQL instance can no longer keep up

Your application writes are climbing, and the primary MySQL instance in your unsharded Vitess keyspace is hitting CPU or disk limits. Adding read replicas helps read throughput, but it does not reduce the write load on the primary or slow storage growth. Backups take longer, and schema changes become risky. You need to shard, but scheduling downtime is not an option.

Useful takeaway: Vitess can split a shard while it stays online. Data is copied to new child shards while the parent continues accepting reads and writes. Once replication catches up, VTGate routes traffic to the children and the parent is removed. You should consider this path when a single MySQL instance is the write or storage bottleneck, not simply because sharding is available. Sharding adds routing complexity, so confirm the bottleneck first.

Terms you need before splitting

• Keyspace: the logical database in Vitess, such as customer.
• Shard: a slice of a keyspace defined by a key range. An unsharded keyspace has one shard, usually named 0 with key range -.
• Tablet: the Vitess process that runs alongside a MySQL instance and handles replication, queries, and tablet type changes.
• VTGate: the query router that sits between applications and Vitess tablets.
• VSchema: the routing map that tells VTGate which sharding key to use and how key ranges map to shards.
• Keyrange: the hexadecimal range of sharding key values owned by a shard, such as -8000000000000000.

How the split and reserve workflow works

Vitess copies data from the parent shard to two new child shards while the parent remains read/write. Replication keeps the children current. When lag is low enough, VTGate begins sending writes and reads for each key range to the matching child. After the switch, the parent shard is removed. This is often called online resharding because the application does not stop.

The workflow depends on the VSchema. If a query omits the sharding key, VTGate cannot determine the destination shard and issues a scatter query to every shard in the keyspace. After a split, scatter queries hit more shards and can increase latency and load.

Worked example: splitting shard 0 of the customer keyspace

Run vtctldclient from an administration host with network access to the Vitess vtctld API. You need cluster-level permissions to read and modify topology. Exact flags vary by Vitess version; treat the following as a structural example, not a copy-paste command.

1. Inspect the current keyspace.
   vtctldclient --server vtctld.example.com:15999 GetKeyspace customer
   Expected: one shard named 0 with key range -.
2. Apply the VSchema on a non-production cluster first.
   Create vschema.json mapping the sharding column, for example customer_id, to the keyspace's sharding key. Then:
   vtctldclient --server vtctld.example.com:15999 ApplyVSchema customer vschema.json
   Run sample SELECT statements through VTGate and confirm they route to the expected shard.
3. Start the split.
   vtctldclient --server vtctld.example.com:15999 SplitShard customer/0 -8000000000000000 8000000000000000-
   This creates child shards and begins copying data. The parent stays available.
4. Monitor replication lag.
   vtctldclient --server vtctld.example.com:15999 ShardReplicationPositions customer/-8000000000000000
vtctldclient --server vtctld.example.com:15999 ShardReplicationPositions customer/8000000000000000-
   Watch lag until it is consistently near zero before cutting traffic over.
5. Confirm the result.
   vtctldclient --server vtctld.example.com:15999 GetKeyspace customer
   Expected after cutover: child shards exist, the parent shard 0 is gone, and VTGate routes by key range.

Risks during the copy phase: increased MySQL read load on the parent, network traffic between tablets, and disk consumption on target tablets. Verify that target tablets have enough free disk space for the full shard copy plus growth before starting.

Trade-offs and limitations

• Time: a split can take hours or days depending on data size and replica lag. Monitor tablet health and replication position throughout.
• Scatter queries: any query without the sharding key fans out to all shards. After the split, those queries touch more nodes and can become slower.
• Routing errors: an incorrect VSchema or invalid key range can route writes to the wrong shard or cause failures. Test on staging before production.
• Rollback: because resharding changes cluster state, you can abort before the traffic switch by stopping the copy and removing child shards. After traffic has switched, reverting requires a reverse reshard or restoring from backup, so treat the cutover as a deliberate decision.

Practical check before you cut over

Run a representative query through VTGate that includes the sharding key and confirm the response comes from only the expected child shard. Then run a query without the sharding key and observe whether it scatters to all shards. If scatter queries are common in your workload, plan to rewrite them or accept the extra load after the split.

Closing

Split a Vitess shard when a single MySQL instance is genuinely constrained by writes or storage, and when your team can absorb the operational complexity of sharded routing. Test the VSchema and the split on a staging cluster, monitor replication lag during the copy, and confirm child shard routing before removing the parent.