Impact of find_unused_parameters on DDP gradient synchronization efficiency
In PyTorch Distributed Data Parallel (DDP), the find_unused_parameters flag is used to handle models where some parameters do not contribute to the loss in every forward pass. When enabled, DDP must traverse the autograd graph to identify these unused parameters and exclude them from the all-reduce synchronization process. While this ensures correctness for
pytorch