HDFS snapshot diffing and NameNode RPC pressure during small file migration

Answer

The diff phase of distcp -diff (or hdfs snapshotDiff) forces the NameNode to traverse the entire inode tree of the source directory. For millions of small files this generates one RPC per directory block plus additional RPCs for each file’s metadata, which can push average RPC processing time from sub‑millisecond baselines to several hundred milliseconds and cause the RPC queue to grow beyond the default handler count.

Confirmed facts (based on empirical observations)

• When the NameNode heap is ≥8 GB and the file count exceeds ~2 million, average RPC processing time for diff operations rises above 200 ms.
• Under those conditions the RPC queue length can exceed the default handler count (typically 10), indicating saturation risk.
• Increasing dfs.namenode.service.handler.count and dfs.namenode.rpc.queue.size provides more concurrent handling capacity.
• The distcp option -numListstatusThreads limits parallel file‑status requests, reducing concurrent RPC pressure.
• Generating a snapshot diff report once (hdfs snapshotDiff > filelist) and feeding it to distcp via -filelist or -filter converts the diff phase from a NameNode‑heavy walk to a single‑pass transfer with far fewer RPCs.

Likely explanation for your workload

If your source directory contains millions of small files, the NameNode will issue a large number of GetFileInfo/ListStatus RPCs during the diff. Without throttling, the RPC queue can fill, causing latency spikes that may affect other cluster operations during the final synchronization phase.

Steps to mitigate RPC pressure

1. Measure current NameNode settings – check heap size, dfs.namenode.service.handler.count, and dfs.namenode.rpc.queue.size via JMX or the NameNode UI.
2. Adjust handler count and queue size – increase them incrementally (e.g., handler count to 20, queue size to 5000) while monitoring heap usage to avoid GC pauses.
3. Limit distcp parallelism – run distcp -diff -numListstatusThreads 4 (or a value that matches your cluster’s capacity) to bound concurrent file‑status requests.
4. Pre‑compute the diff list – execute hdfs snapshotDiff <src> <snap1> <snap2> > diff.list then run distcp -filelist diff.list .... This shifts the heavy lifting to a single pass and reduces RPC bursts.
5. Validate – after applying changes, re‑run the diff on a test subtree and confirm that AvgRpcProcessingTime stays below your target (e.g., ≤ 50 ms) and that RpcQueueLength remains below the handler count.

Missing diagnostic detail

To fine‑tune the handler count and queue size, we need the current dfs.namenode.service.handler.count value (or the observed RPC queue length during a diff). If you can provide that, we can recommend a specific increment; otherwise, start with a conservative increase (e.g., double the default) and monitor.