Kafka to Spark Structured Streaming: can checkpoint state survive a zero-downtime migration?

Migration goal

Consider a small Structured Streaming job that reads from Kafka and writes to a relational sink. The plan is to move it to a newer Spark 3.x release, possibly with a revised aggregation, without pausing the pipeline. The open decision is whether the existing checkpoint directory can carry the query across that boundary, or whether cutover needs a parallel run with a fresh checkpoint.

Constraints and uncertainty

• Checkpoint state — offsets, state store data, and metadata — is tied to the query's logical plan, so edits to stateful operators are generally treated as a new query.
• Checkpoint compatibility across Spark versions is release-note-dependent and should be confirmed for the specific target version before relying on reuse.
• A shadow run consumes the topic twice and can duplicate sink writes unless the sink is idempotent or transactional.
• End-to-end exactly-once depends on the sink's contract, not on Spark alone.

Open questions

1. For a job this small, is checkpoint reuse across the upgrade realistic, or is a fresh checkpoint plus a bounded reprocessing window the safer default?
2. Which plan changes — watermark, aggregation, output mode — are documented as invalidating checkpoint state and forcing a restart?
3. What sink-side contract, such as idempotent upserts versus transactional commits, keeps cutover duplicate-free during a parallel run?