RuntimeError: NCCL timeout during DDP gradient synchronization

Question

PyTorch Distributed Data Parallel (DDP) utilizes collective communication primitives via the NCCL backend to synchronize gradients across multiple GPU ranks. When a network partition or process failure occurs during the backward pass, the system typically triggers a timeout error that halts the entire training job.

The primary challenge is the lack of a native, idempotent retry mechanism for these collective operations. If a synchronization call fails partially, there is no built-in method to recover the specific failed operation without restarting the process group or reloading from a global checkpoint, as manual retries risk creating weight divergence between ranks.

How can a failed collective communication call be recovered without restarting the entire training epoch?
  What is the documented behavior for maintaining model consistency across ranks if a subset of processes recovers from a timeout while others do not?

Tasadduq Burney · Answer

The Hard Truth: No In-Place Retry for NCCL Timeouts

When NCCL times out during DDP gradient synchronization, PyTorch tears down the entire process group. This is by design—not a bug. The collective operations (all-reduce, all-gather) are atomic across all ranks, and a timeout on any single GPU means the operation cannot be considered complete on any rank.

Why Manual Retries Fail

Attempting to catch the RuntimeError and call dist.all_reduce again is unsafe because:

The process group state is already destroyed—subsequent collectives will hang or fail immediately
  Even if the group were intact, ranks that did not timeout would have already applied gradients, causing weight divergence
  Optimizer step state would be inconsistent across ranks

The Only Safe Recovery Path

Use torch.distributed.checkpoint (PyTorch 2.3+) to save sharded checkpoints, then restart from the last known good state. The workflow is:

Checkpoint regularly: Save model, optimizer, and RNG state every N steps
  Enable elastic restart: Use torchrun --rdzv_backend=c10d --max_restarts=3 with TorchElastic
  Handle the exception: Catch the timeout, let TorchElastic restart workers
  Verify continuity: Ensure loss curves resume from checkpointed state

Model Consistency: All or Nothing

PyTorch guarantees consistency only when all ranks either complete the collective successfully or all roll back to the same checkpoint. There is no documented mechanism for partial recovery. If one rank recovers while others restart, parameters will diverge silently—this is not a recovery, it's corruption.

Diagnosis Checklist

Before increasing NCCL_TIMEOUT, investigate the root cause:

Check GPU utilization: OOM can cause one rank to hang during backward
  Verify network health: InfiniBand or Ethernet congestion
  Review NCCL logs with NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=ALL
  Confirm batch size consistency: Uneven batches can cause rank imbalance

Key Parameters

ParameterValueNotes
  
    NCCL_TIMEOUT30-60 minutesIn milliseconds; longer values delay failure detection
    NCCL_BLOCKING_WAIT1Enables blocking waits for debugging
    find_unused_parametersFalseTrue adds extra collectives, increases timeout risk

Bottom Line

For NCCL timeout recovery, treat it as a process-level failure, not an operation-level one. The only recommended approach is checkpoint-based restart with TorchElastic. Any attempt at in-place retry risks silent model corruption across your distributed training run.

RuntimeError: NCCL timeout during DDP gradient synchronization

1 answer