Kafka to Spark Structured Streaming: can checkpoint state survive a zero-downtime migration?

Answer

Checkpoint state cannot be carried over a zero‑downtime migration when either the Spark version changes or the logical plan of stateful operators is altered; reuse is only safe if both conditions are met and the target release explicitly guarantees checkpoint compatibility.

Confirmed facts

• A Structured Streaming checkpoint stores Kafka source offsets, query metadata, and the state‑store snapshot (RocksDB or HDFS‑backed). The state‑store format is versioned and tied to the query’s logical plan and the Spark version that wrote it.
• Spark will refuse to start a query if it detects an incompatible state‑store schema or offset‑log version.
• Changes that invalidate the checkpoint include modifying aggregation expressions, grouping keys, watermark definition, output mode, or adding/removing stateful operators. Stateless filter or projection changes are usually tolerated.
• Cross‑version checkpoint compatibility is not guaranteed; it must be verified in the migration notes for the exact source and target Spark releases.

Likely explanation

When you upgrade Spark or tweak a stateful operator, the internal version of the state store often bumps, making the persisted snapshot unreadable by the new runtime. Even if the on‑disk format stayed identical, a different logical plan changes the expected schema of the state rows, causing a schema mismatch on restart.

Steps for this case

1. Identify the exact source Spark version (e.g., 2.4.x) and target version (e.g., 3.3.x).
2. Consult the migration guide / release notes for those versions to see if checkpoint format is guaranteed compatible.
3. In a staging cluster, copy the production checkpoint directory to a test location.
4. Start the upgraded query pointing at that copy (same query name and checkpoint location).
5. If the query starts and resumes from the last committed offset without a state‑store error, reuse is safe; otherwise abort and plan a fresh checkpoint.
6. If a fresh checkpoint is required, set startingOffsets (or startingOffsetsByTimestamp) to the offset saved in the last commit file to bound reprocessing, run the old and new queries in parallel, validate output parity, then cut over.

Missing diagnostic detail

To decide definitively whether reuse is possible, we need to know the exact source and target Spark versions you intend to use. Without that, we cannot confirm compatibility from the release notes.