Impact of find_unused_parameters on DDP gradient synchronization efficiency

In PyTorch Distributed Data Parallel (DDP), the find_unused_parameters flag is used to handle models where some parameters do not contribute to the loss in every forward pass. When enabled, DDP must traverse the autograd graph to identify these unused parameters and exclude them from the all-reduce synchronization process.

While this ensures correctness for dynamic graphs, it introduces overhead during the backward pass. There is a trade-off between the computational cost of graph traversal and the communication efficiency gained by reducing the number of tensors synchronized across the ProcessGroup backend (such as NCCL).

What is the specific performance penalty associated with enabling find_unused_parameters in large-scale models, and does this overhead scale linearly with the number of unused parameters or the total model depth?