HDFS snapshot diffing and NameNode RPC pressure during small file migration

Migrating data between Hadoop clusters without downtime often relies on HDFS snapshots and incremental copies via distcopy. While this approach aims to maintain point-in-time consistency without locking source directories, the performance of the metadata comparison remains a concern for specific workloads.

When the source directory contains millions of small files, the diff calculation between two snapshots requires extensive metadata traversal by the NameNode. There is uncertainty regarding whether this process can lead to sustained RPC saturation, potentially impacting the stability of the production cluster during the final synchronization phase.

What are the specific performance thresholds for NameNode RPC latency when executing distcopy diff on large-scale directories? Can the diffing process be throttled or optimized to prevent resource exhaustion during an incremental migration?