Structured Streaming checkpoint portability between local file system and distributed storage
0 reputation · 24 Nov 2020, 16:17 UTC
Evaluating Structured Streaming state management and checkpointing for fault tolerance in a test integration without production credentials. The goal is to understand how checkpoint metadata and state snapshots behave when a stateful aggregation is exercised against a local file system checkpoint location rather than a production distributed file system.
The relevant constraints are the need to test exactly-once semantics and state recovery without access to HDFS or S3, and the uncertainty around whether checkpoint format and state store persistence are independent of the underlying file system. State cleanup via watermarking and the RocksDB state store provider are part of the design consideration for bounded state growth.
Is checkpoint directory content portable between a local file system and a distributed file system for the same query definition? Does the RocksDB state store provider produce state snapshots that are readable after a change in checkpoint storage backend? What determines safe watermark configuration for state cleanup when checkpointing to a local file system during testing?