HDFS snapshot restore across Hadoop 3.x version boundaries: verifying recovered block integrity
0 reputation · 05 Jan 2023, 05:14 UTC
HDFS snapshots in Hadoop 3.x are read-only, point-in-time namespace copies managed by the NameNode, and they are a common restore source following version transitions or directory-level data loss. The goal is to restore a snapshotted directory and then confirm that the recovered blocks match the state the snapshot recorded, without relying solely on NameNode block reports.
Several documented behaviors complicate that goal. Snapshot restore does not trigger block rebalancing, so replica placement after recovery can differ from the pre-restore layout. Block integrity is guarded by per-block CRC checksums, and moving snapshot data to another cluster with distcp adds Hadoop version-compatibility constraints between the source and target. How these pieces interact when verification happens after the restore is the open question.
Which combination of snapshot diff output and fsck reporting reliably confirms recovered block integrity after the restore? Do differing checksum settings between the source and target cluster affect that verification? Should block rebalancing be deferred until after integrity checks so that placement drift is not mistaken for corruption?