Spark Structured Streaming checkpointing and HDFS snapshot integration for backup restoration

Recovery Strategy: WAL vs. Snapshots

Apache Spark Structured Streaming's internal checkpointing and Write-Ahead Log (WAL) are sufficient for fault tolerance (recovery from driver or executor crashes), but they are not a substitute for disaster recovery (recovery from directory deletion or filesystem corruption). You should rely on WAL replay for routine failures and implement periodic HDFS snapshots as a backup mechanism.

Internal WAL vs. External Snapshots

The internal WAL provides exactly-once guarantees by recording offsets and state changes before they are committed. If the checkpoint directory is intact, Spark simply replays the WAL upon restart to resume processing. However, because the WAL resides within the checkpoint directory, any catastrophic loss of that directory renders the WAL useless.

HDFS snapshots provide a point-in-time recovery point. While they add storage overhead and metadata pressure on the NameNode, they protect against accidental hdfs dfs -rm -r commands or filesystem-level corruption that internal Spark mechanisms cannot resolve.

Implementation Steps for Backup Restoration

1. Configure Checkpointing: Ensure checkpointLocation is set to a durable HDFS path accessible by all nodes.
2. Schedule Snapshots: Create HDFS snapshots of the checkpoint directory at defined intervals (e.g., daily or hourly). Note: To avoid inconsistent states, snapshots should ideally occur between batch intervals.
3. Restore Process: In the event of directory loss:
   • Restore the checkpoint directory from the latest HDFS snapshot.
   • Restart the Spark streaming application.
   • Spark will replay the WAL files present in the restored snapshot to determine the last committed offset.

Operational Trade-offs and Verification

Metric	WAL-Only Recovery	Snapshot Restoration
Recovery Time	Fast (immediate replay)	Slower (restore time + replay)
Storage Cost	Minimal (active logs)	Higher (point-in-time copies)
Data Integrity	Exactly-once (if dir exists)	Depends on snapshot consistency

Version-Specific Considerations

While the core logic remains consistent, Spark 3.x offers improved WAL handling and log cleanup compared to 2.x. For Spark 2.x users, verify spark.streaming.driver.writeAheadLog.closeTimeout to ensure logs are flushed correctly before the driver shuts down.

Verification Command: To verify the current snapshot state of your checkpoint directory, use:

hdfs dfs -ls /path/to/checkpoint/.snapshot

Diagnostic Requirement: To refine this recommendation, please specify if your output sink is idempotent. If the sink is not idempotent, restoring from an older snapshot may cause duplicate data processing during the WAL replay of the restored state.