Unspecified tensor ordering in torch.distributed.all_gather causes silent aggregation failures
PyTorch's torch.distributed.all_gather collective returns tensors from all ranks, but the order of the returned list is unspecified. This is a documented failure condition for any workflow that assumes a fixed rank-to-index mapping, including deterministic gradient aggregation and parameter synchronization across processes. Because the ordering is not guaran
pytorch