Vitess Binary Rollback and MySQL State Interoperability

Short Answer

A binary-only downgrade of vttablet is viable only if the failed upgrade left no incompatible on-disk or topology state behind: the _vt sidecar schema is unchanged (or changed only additively), topology records still parse under the old binary, no vreplication or online-DDL workflows were created or advanced under the new version, and MySQL itself was not upgraded. If any of those conditions fails, the safe path is restoring the pre-upgrade backup and replaying binlogs to recover writes accepted during the upgrade window.

Why the N/N-1 Guarantee Doesn't Cover Rollback

Vitess's version compatibility promise is about mixed-version operation during a rolling upgrade — an N-1 vtgate talking to an N vttablet, and vice versa. It is not a promise that an older binary can correctly interpret state written by a newer one. That asymmetry is the core of your question:

• Topology records. Vitess stores tablet, keyspace, and shard records in the topo server as serialized protobuf (typically JSON-encoded). If the newer version added fields or rewrote records, the older vttablet may fail to load them at startup. The failure mode here is usually loud — the tablet won't come up healthy — which is actually the better outcome.
• The _vt sidecar database. This is the subtler risk. Purely additive migrations (new tables, new columns) are often tolerated because vttablet queries explicit column lists rather than SELECT *. But semantic changes — altered vreplication state values, changed position or timestamp encodings, resumable-DDL redo tables — can make the N-1 binary misbehave silently: it starts, serves, and quietly does the wrong thing.
• MySQL itself. Vitess does not normally rewrite MySQL system tables; its footprint is the _vt database plus some global variables and users. But if your upgrade bundled a MySQL minor/major upgrade, the old mysqld may be unable to open the data directory at all — MySQL in-place downgrades are heavily restricted and are a separate problem from the Vitess rollback.

Decision Criteria

Binary rollback alone is sufficient when all of the following hold:

1. The _vt schema is identical to its pre-upgrade state, or differs only additively.
2. Topo records are unchanged or still parse under N-1 (verify by dumping them, not by hoping).
3. No vreplication streams, online DDL, or other durable workflows were created or advanced while the new binary ran.
4. MySQL was not upgraded as part of the same change.

In that case: stop the new vttablet, start the N-1 binary, and verify — tablet health via vtctldclient, replication lag, and that any existing vreplication streams resume. Do not treat "process is up" as proof of correctness; check replication and throttler behavior explicitly.

A full restore is required when any of these is true:

• The sidecar schema was migrated non-additively.
• Topo data was rewritten in a way N-1 rejects.
• Vreplication streams exist that the old binary cannot resume.
• MySQL was upgraded and the old server can't open the datadir.

Restore the pre-upgrade backup — not a backup taken by the newer version, whose metadata or format N-1 may not understand — then replay binlogs from the backup's position forward to recover writes accepted during the upgrade window.

A Middle Path Worth Checking First

If the rolling upgrade was interrupted mid-fleet, any replica still running N-1 with intact state can serve as the source to resync the affected tablet (re-parent or restore-from-peer depending on your setup). This avoids both the full restore and the data-loss trade-off, and is often the fastest recovery when only one or two tablets got as far as writing new state.

Verifying What Actually Changed

Before choosing a path, establish facts rather than inferring them from how far the upgrade appeared to get:

• Diff the sidecar schema: SHOW CREATE TABLE for each _vt table against a pre-upgrade capture or a still-on-N-1 peer.
• Dump and diff the topo records (etcdctl, zkCli, or Consul, depending on your topo implementation) for the Vitess key prefix.
• Read the release notes and upgrade guide for your exact version pair — sidecar, topo, and vreplication changes are version-pair-specific, and that documentation is the authoritative word on whether your downgrade is clean.

Caveats

The specific viability of any rollback depends on the exact N to N-1 pair and on what your upgrade actually executed; treat the criteria above as the decision framework, and confirm against the version-specific release notes. If you can share which two versions are involved and whether the upgrade reached the point of running sidecar migrations, the recommendation can be narrowed further. For next time: take a fresh backup and a topo export immediately before upgrading, quiesce vreplication workflows, and keep the vttablet upgrade window short — then a rollback never has to trade data loss against consistency.