Changing Vitess Schemas Without Taking Shards Offline

Adding a column to a sharded MySQL table in production usually means picking a maintenance window. In Vitess the problem is the same, but the routing layer gives you a safer way to evolve the schema: let vtctl OnlineSchemaChange build a new table, copy data in the background, and swap metadata so VTGate keeps routing reads and writes without an application change.

The useful takeaway is that you can treat schema evolution as a data migration with a metadata cutover, not an in-place ALTER. That matters because Vitess tablets are independent MySQL instances and VTGate routes by keyspace and shard metadata. If the metadata swap is done correctly, clients see a continuous service.

Why OnlineSchemaChange fits Vitess sharding

VTGate routes queries to the appropriate tablet using keyspace and shard metadata. That indirection means the logical table name can be remapped to a physical table during a migration.

vtctl OnlineSchemaChange implements the change by creating a new table with the desired schema, copying existing rows, and then swapping metadata to make the new table the active one. The old table is kept for rollback. Because each tablet is isolated, the copy and swap can proceed per shard with independent failover and replication handling.

Running a schema change with vtctl

Run vtctl from a machine with access to the Vitess control plane and permissions to submit schema changes for the target keyspace. The command is typically executed against a topology server and requires the operator to know the keyspace, table, and exact ALTER statement.

A representative invocation looks like:

vtctl OnlineSchemaChange \\
  -keyspace <keyspace> \\
  -table <table> \\
  --alter \"ADD COLUMN email_verified tinyint(1) NOT NULL DEFAULT 0\" \\
  --throttle-threshold-bytes 104857600

Placeholders:

• <keyspace>: the Vitess keyspace name, e.g. commerce
• <table>: the logical table name, e.g. user

Meaningful options to review first:

vtctl OnlineSchemaChange --help

That lists available flags for throttling, concurrency, and dry-run behavior. Throttling is important because the data copy generates read and write load on source tablets and replication traffic to replicas.

Expected checks during execution:

• Confirm the new shadow table appears in the tablet schema and the copy job progresses per shard.
• Monitor replication lag on tablets while copying large tables.
• Verify VTGate continues to route queries to the original table until the metadata swap is signaled.

Risks:

• Large tables increase replication lag during the copy phase.
• vtctl and tablet versions must be compatible. Mismatched versions may cause schema application failures.
• The ALTER must be supported by the online change path. Complex rewrites may require manual review.

Verification and trade-offs

After the change completes, verify routing and schema consistency:

• Execute a read on a test keyspace and confirm VTGate routes queries to the new table metadata.
• Check replication status using vtctl TabletExplain and tablet health endpoints after the change.
• Compare row counts between old and new tables on a sample shard before finalizing.

Limitations to plan for:

• Copying data is I/O intensive. On very large shards, expect increased latency and replication lag.
• OnlineSchemaChange does not change application sharding logic, but it does change the physical table name temporarily. Any direct MySQL connections bypassing VTGate will not see the swap.
• Rollback is possible while the shadow table exists, but once the metadata is swapped the old table is retained for a retention window. Deleting it prematurely removes the rollback path.

An actionable way to proceed is to run the change first on a non-critical keyspace or a single shard, observe replication lag and VTGate routing, then promote to production shards with throttling tuned to your workload.