Cassandra 3.11 to 4.0 Rolling Upgrade: Where Does the Snapshot Rollback Window Actually Close?

Question

A rolling upgrade from Cassandra 3.11 to 4.0 needs a rollback plan, and the unresolved question is where that plan stops working if the upgrade fails midway. The documented per-node sequence (drain, snapshot, then restart on 4.0) leaves pre-upgrade snapshots as hard links to existing SSTables, and the 3.11 binary can neither read 4.0-format SSTables nor replay 4.0 commitlog segments.

That suggests the real boundary is the first write a 4.0 node accepts — its flushed memtables and commitlog segments are immediately new-format — not the point where nodetool upgradesstables or compaction rewrites older files. The two events are often conflated, muddying the upgradesstables-timing decision.

Constraints: the mixed-version window rules out DDL and repairs; coexisting old- and new-format SSTables roughly double data-directory usage; and 4.0's changed TTL expiration handling (the 2038 local-expiration overflow) adds caveats for long-TTL tables during the window. Flags and limits vary by release, so NEWS.txt for the exact pair needs confirming.

Is the practical rollback limit per node the first post-upgrade write, with upgradesstables timing affecting only conversion of pre-existing files?
For a multi-day rollback window, is deferring upgradesstables a supported posture, or does it only prolong mixed-format disk use without real rollback benefit?
Does snapshot restore on a partially upgraded cluster require reverting every node that already accepted 4.0 writes, even where old-format SSTables remain on disk?

Tasadduq Burney · Answer

AnswerThe practical rollback limit for a node is the moment it first accepts a write while running Cassandra 4.0 and that write is flushed to a memtable or committed to the commitlog. At that point the node has created new‑format SSTables (nb) and/or new‑format commitlog segments that the 3.11 binary cannot read, so rolling back to 3.11 would lose any data written after that point.Confirmed factsCassandra 4.0 writes SSTables in the “nb” (big‑format) version and new commitlog segments; the 3.11 binary cannot parse either format.Running nodetool upgradesstables only rewrites pre‑existing 3.x‑format SSTables; it does not create the incompatibility.The upgrade documentation explicitly advises postponing upgradesstables until you are certain you will not roll back, confirming that deferring it preserves the rollback option.Likely explanation (needs verification)The exact boundary can be viewed as “node restarted on 4.0 and has accepted traffic”, because the commitlog is created at startup and the first flushed memtable produces the incompatible files. Individual write timing does not move the boundary.Steps for this caseDrain the node (nodetool drain).Take a snapshot (nodetool snapshot -t pre-upgrade).Stop the node, install Cassandra 4.0, start it.Do not run nodetool upgradesstables yet.Monitor writes; if you need to roll back before any write has been flushed, stop the 4.0 node, remove its data and commitlog directories, restore the snapshot via hard links, and start the node with Cassandra 3.11.If writes have already occurred, accept that data written after the first flush is lost; you may export it before reverting.Missing diagnostic detailTo confirm version‑specific notes (e.g., TTL expiration changes, any extra flags), please provide the exact 3.11.x and 4.0.x versions you are using so I can check the corresponding NEWS.txt.

Cassandra 3.11 to 4.0 Rolling Upgrade: Where Does the Snapshot Rollback Window Actually Close?

1 answer

Answer

Confirmed facts

Likely explanation (needs verification)

Steps for this case

Missing diagnostic detail

Your perspective belongs here.

0 question comments