Structured Streaming checkpoint portability between local file system and distributed storage

Answer

Yes – the checkpoint directory written by Structured Streaming is portable between a local filesystem and a distributed filesystem (HDFS, S3, etc.) for the same query definition, and the RocksDB state store files can be read after moving the directory, provided the Spark/Structured Streaming version and dependent libraries match.

Confirmed facts

• Checkpoint metadata (offsets, commit logs) and state snapshot files are written in a format that depends only on the file paths, not on the underlying filesystem.
• When using the RocksDB state store provider, the state store directory contains pure binary RocksDB files (LOG, MANIFEST-*.ldb, *.sst, etc.) that are readable by any Structured Streaming job that points its checkpointLocation to that directory, regardless of whether the directory resides on local disk or a distributed store.
• Watermark safety for state cleanup is determined solely by the event‑time column and the watermark delay you specify; the delay must exceed the maximum expected out‑of‑orderness in the data and is independent of the checkpoint storage backend.

Likely explanation

The portability stems from Structured Streaming treating the checkpoint location as an opaque URI; it writes files using Hadoop’s FileSystem API, which abstracts the underlying storage. RocksDB stores state as platform‑independent SST and log files, so moving the directory does not alter the binary format. Watermark logic lives in the query plan and does not interact with the filesystem.

Steps to verify portability in your test

1. Run your stateful aggregation query with RocksDB state store, checkpointing to a temporary local directory (e.g., file:///tmp/ss-checkpoint). Process a few batches, then stop the query.
2. Copy the entire checkpoint directory to the target distributed storage (e.g., hdfs://namenode:8020/ss-checkpoint or an S3 bucket) using distcp, aws s3 sync, or a simple cp if mounted.
3. Start a new Structured Streaming job with the exact same query definition, pointing checkpointLocation to the copied URI.
4. Confirm that the job resumes from the same offset and restores state without errors (check logs for Resuming from checkpoint and metric numInputRows matching the previous run).
5. Inspect the directory to verify the presence of RocksDB files (LOG, MANIFEST-*.ldb, *.sst) and metadata files (offsets, commit).

Watermark configuration for local testing

Choose a watermark delay that reflects the latency you observe in the test data stream. For example, if your test source emits events with at most 5 seconds of out‑of‑orderness, set withWatermark('eventTime', '5 seconds'). The delay does not need to be adjusted when switching between local and distributed checkpoints.

Missing diagnostic detail

To give a definitive recommendation, please confirm the Spark/Structured Streaming version (e.g., 3.5.0) used for both the original checkpoint‑writing job and the job that reads the copied checkpoint. Version mismatches can cause deserialization errors even though the filesystem is the same.