QuasarDB schema change rollback: backup vs dual‑write strategy

Short answer

If rollback must be possible without losing any writes, use a dual-write strategy with an application-controlled cutover. Backup/restore is only acceptable if you can tolerate downtime and the loss of every write made after the snapshot — QuasarDB restore returns the cluster to the snapshot state and does not merge or retain post-snapshot writes. Those writes are simply gone unless you re-apply them yourself.

Why backup/restore is the weaker rollback tool here

Because QuasarDB tables are immutable after creation, a schema change means a new table, a backfill, and a cutover. A cluster backup taken before the change gives you a clean revert point, but restore is all-or-nothing and time-bound:

• Any writes to the new table (or anywhere else in the cluster) after the snapshot are discarded on restore. They are not merged.
• The longer the new schema runs before you decide to roll back, the larger the data-loss window becomes.
• Restore typically implies downtime or at least a service disruption while the cluster is reverted.

Backup/restore is therefore reasonable only for a fast fail-fast validation: take the snapshot, cut over, validate within minutes, and restore immediately if something is wrong — accepting that the small window of writes must be replayed from an upstream source (queue, WAL, application log) or sacrificed.

The dual-write pattern that preserves rollback

The safer pattern keeps the old table as the system of record until the new one is proven:

1. Create the new table with the desired column set.
2. Backfill existing data from old to new, recording a high-water mark (timestamp or sequence) so you know where backfill ends and live writes begin.
3. Dual-write: the application writes to both tables. Reads still come from the old table.
4. Verify parity: compare row counts and aggregate checksums between the tables over matching time ranges before trusting the new one.
5. Read cutover: point reads at the new table while continuing dual-writes. Keep this reversible.
6. Write cutover: stop writing to the old table only after the new table has run cleanly for your confidence window.
7. Retire the old table later — a TTL can expire it automatically after a predetermined window, which avoids a manual drop while cutover might still be in progress.

Rollback at any point before step 6 is trivial: stop writing to the new table and keep using the old one, which never stopped receiving writes. No data is lost because the old table stayed current.

Handling divergence in dual-write

The real cost of dual-write is divergence when one write succeeds and the other fails. Practical mitigations:

• Idempotent writes. Make writes keyed and retryable so a failed write to one table can be replayed without duplication.
• Failure queue. When a write to either table fails, push the operation onto a retry queue (or dead-letter log) instead of silently dropping it. Alert on queue depth — that is your divergence signal.
• Periodic reconciliation. Run a scheduled comparison of counts and per-bucket checksums over recent time windows. QuasarDB's aggregation queries make count/sum comparisons cheap; mismatches trigger a targeted re-backfill of the affected range.
• Ordering discipline. Write to the table that is currently the read source first, so a partial failure never leaves readers ahead of the source of truth.

Verification before you commit

Before dropping dual-write, confirm: row counts match per time bucket, aggregate checksums match, application error rates are flat, and a read-only test against the new table returns expected results. One detail that changes this recommendation: if your writes arrive through a replayable stream (e.g., a durable queue), backup/restore becomes much more attractive because post-snapshot writes can be replayed after restore — if that is your setup, say so, since it simplifies the whole design.