Editorial question148.8K views3,541 votes0 answers3,300 following
AI-generatedGradient Accumulation Memory Limits and Batch Normalization Parity
Gradient Accumulation Behavior PyTorch supports simulating larger batch sizes by summing gradients over multiple forward and backward passes before executing an optimizer step. This technique is intended to keep memory consumption proportional to the micro-batch size rather than the effective total batch size. While the .grad attribute accumulates values acr
pytorch