vtctl Rolling Upgrade vschema Consistency Gap

1. When ResetSchema Is Safe With Mixed Tablet States

Vitess’s ResetSchema will only succeed when every tablet in the keyspace reports a Healthy status and has finished any pending schema changes. The command checks two conditions on each tablet:

• Tablet status must be Serving and Healthy in vtctld’s health table.
• No pending DDLs (vtctl GetTabletSchema should return the same schema as the live MySQL instance).

If any tablet is in Pending, Unhealthy, or still applying a DDL, ResetSchema aborts to avoid partial routing updates. In practice, you should run:

vtctl GetTabletSchema --all

and confirm that every result matches the expected schema and that vtctld shows Healthy for all tablets before invoking ResetSchema.

2. Selectively Reverting an etcd vschema Revision

etcd keeps a revision history for each key. You can restore a prior revision without restarting the topology by using etcdctl:

# View revision history (requires etcdctl v3.5+)
# Replace  with your vschema key, e.g., /vitess/vschema/keyspace
etcdctl get /vitess/vschema/ --rev=0 --print-value-only | less

# Fetch a specific revision
etcdctl get /vitess/vschema/ --rev= --print-value-only > backup.json

# Restore that revision
etcdctl put /vitess/vschema/ @backup.json

After restoring, run vtctld --vschema_sync to push the corrected vschema to all tablets. Note that this bypasses Vitess validation, so the restored revision must match the live MySQL schemas to prevent routing errors.

3. Detecting and Resolving vschema Drift After a Failed Upgrade

1. Verify tablet health: vtctl GetTabletSchema --all and ensure all tablets report Healthy.
2. Diff vschema: vtctld --vschema_diff compares the etcd vschema against each tablet’s live schema. Review any mismatches.
3. Identify drift shards: The diff output lists shards where the metadata diverges.
4. Reconcile: If the etcd revision is correct, use vtctld --vschema_sync to push the live schema back into etcd. If the live schema is correct, restore the appropriate etcd revision as described above.
5. Re‑run ResetSchema: Once all tablets are healthy and schemas are aligned, execute vtctl ResetSchema.
6. Validate routing: Send a test query through vtgate to confirm that routing tables have been rebuilt correctly.

These steps avoid data loss because they never drop or truncate tables; they only adjust routing metadata and ensure consistency between etcd and the MySQL instances.

4. Quick Diagnostic Question

Before proceeding, could you confirm whether any tablets still report a Pending DDL status after the upgrade attempt?