HDFS snapshot restore across Hadoop 3.x version boundaries: verifying recovered block integrity

Answer to the question

When you restore a snapshot on a Hadoop 3.x cluster you can be confident that the recovered data is byte‑for‑byte identical to the original snapshot by combining two tools:

• Snapshot diff – gives you the list of files that should exist after the restore.
• fsck –verifyBlockChecksum – re‑computes the CRC for every block and reports any mismatch.

Because snapshot diff does not expose block IDs, the verification is performed at the file level: you confirm that every expected file is present and that none of its blocks are reported as corrupt. Replica placement changes caused by the lack of automatic rebalancing after a restore do not influence the checksum check, so a subsequent balancer run will not produce false corruption reports.

Why this combination works

• The snapshot diff lists file paths and the operations performed between two snapshots (CREATE, DELETE, RENAME). By filtering for files that exist in the target snapshot you get the exact set of files that should be present after the restore.
• The hdfs fsck command with -verifyBlockChecksum walks the namespace, reads each block’s stored CRC, recomputes it locally, and flags any block that does not match.
• Both tools operate on the same namespace; changes in replica placement or block replication factor do not affect the CRC comparison.
• Checksum algorithm mismatches between source and target clusters can cause fsck to report corruption even though the data is identical. Therefore the checksum type must be identical on both clusters before running the check.

Practical verification steps

1. Capture the expected file list from the snapshot diff:

   hdfs dfs -getSnapshotDiff /data/snap_dir snap1 snap2 > /tmp/diff.txt
   # Keep only the lines that indicate files that exist in snap2 (CREATE or RENAME to snap2)
   # Example: parsing with awk
   awk '/CREATE/ || /RENAME/ {print $2}' /tmp/diff.txt > /tmp/expected_files.txt
   

2. Run fsck with checksum verification:

   hdfs fsck /data/snap_dir -files -blocks -locations -verifyBlockChecksum > /tmp/fsck.txt
   

   The output contains one line per block, e.g.:

   Block 123456 (size=128 MB) : OK
   

3. Cross‑check file presence and block status:

   # Extract block IDs for each file from fsck
   awk '/^Block / {print $2}' /tmp/fsck.txt > /tmp/fsck_blocks.txt
   # Verify that every file in expected_files.txt has at least one block and none are marked CORRUPT
   # (simple sanity check – detailed mapping requires the block‑to‑file table from the NameNode)
   

4. Identify corrupt files explicitly:

   hdfs fsck /data/snap_dir -list-corruptfileblocks > /tmp/corrupt_files.txt
   

   Any file listed here indicates a block that failed the CRC check and needs further investigation.

5. Check checksum type and block size compatibility:

   hdfs getconf -confKey dfs.block.checksum.type
   hdfs getconf -confKey dfs.block.size
   

   Compare the output against the source cluster. If they differ, adjust the target cluster’s configuration (edit core-site.xml or use hdfs dfsadmin -setSafeMode followed by a restart) so that both clusters use the same values before re‑running fsck.

6. Optional: verify placement drift:

   hdfs fsck /data/snap_dir -files -blocks -locations > /tmp/placement_before.txt
   # Run a balancer if desired
   hdfs balancer -threshold 10
   hdfs fsck /data/snap_dir -files -blocks -locations > /tmp/placement_after.txt
   

   Differences between the two files show only replication placement changes; they do not indicate corruption.

7. Rebalance after verification (if required):

   hdfs balancer -threshold 10
   

   This step does not affect block checksums, so it can safely be performed after the integrity check.

Things that can cause false positives

• Different dfs.block.size values between clusters cause the same file to be split into a different number of blocks, which may lead to missing or extra block entries in fsck output. Verify that the block size is identical before running the check.
• Different dfs.block.checksum.type values (e.g., CRC32 vs CRC32C) will make fsck report corrupt blocks even when the underlying data is the same. Align the setting on the target cluster before verification.
• Running fsck on very large directories can consume significant CPU and I/O. In such cases consider running it on a subset of the directory or using the -files flag to limit the depth.

Diagnostic detail needed

To fine‑tune the verification, please provide the block size and checksum type of the source cluster:

hdfs getconf -confKey dfs.block.size
hdfs getconf -confKey dfs.block.checksum.type

Summary

By first extracting the list of files that should exist after a snapshot restore, then running hdfs fsck with -verifyBlockChecksum and checking for corrupt blocks, you can reliably confirm that the restored data matches the original snapshot. Ensure that the target cluster’s block size and checksum algorithm match the source; if they differ, adjust the configuration before verification. Replica placement drift caused by the lack of automatic rebalancing after a restore will not be mistaken for corruption, so you can safely rebalance after the integrity check.