Which GradScaler settings matter when choosing float16 over bfloat16 for AMP training?

Answer first

On Ampere-class GPUs with a recent PyTorch release, bfloat16 autocast is normally used without GradScaler, while float16 autocast is used with GradScaler. The scaler knobs that matter are init_scale, growth_interval and backoff_factor. In practice the defaults are sufficient for most models; changes are only justified when the scale trajectory shows repeated backoff or very slow recovery.

Confirmed facts, version-assumed

Assumption: PyTorch 2.x with device-agnostic autocast torch.amp.autocast and torch.amp.GradScaler. API names moved from torch.cuda.amp to torch.amp across releases, so confirm the constructor defaults for your exact version.

• bfloat16 has an 8-bit exponent like float32 and 7-bit mantissa. Gradient underflow/overflow is rare, so PyTorch docs and common practice pair bf16 autocast with no scaling.
• float16 has ~5-bit exponent, max ~65504, min normal ~6e-5. Loss and gradients are scaled up before backward and unscaled before the optimizer step. GradScaler skips steps that contain inf/NaN.
• GradScaler defaults in recent releases: init_scale = 65536, growth_interval = 2000, backoff_factor = 0.5. The scaler doubles the scale after growth_interval consecutive clean steps and multiplies by backoff_factor on inf/NaN.

Likely explanation for tuning

There is no documented per-model formula for init_scale. Empirically, a reasonable heuristic is to start near the largest power of two that does not immediately produce infs on the first few steps. The scaler self-corrects quickly, so the default rarely needs changing.

growth_interval controls how fast the scale recovers. Lowering it, e.g. 100-500, lets models with frequent early instability regain scale faster, at the cost of more risk of overshooting. Raising it makes recovery more conservative.

backoff_factor controls how aggressively the scale drops on an inf/NaN. The default 0.5 is a compromise. Frequent backoff usually signals genuine numerical instability — loss spikes, exploding gradients, or an op that overflows in fp16 — rather than a tuning problem. Tuning knobs only change reaction speed, they do not fix instability that persists in float32.

When to prefer float16 + scaling over bfloat16

When both dtypes are supported, bf16 is simpler and usually equally fast on Ampere+. Reasons to still choose float16 + scaler:

• Higher mantissa precision: 10 bits vs 7 bits for bf16, which can matter for sensitive reductions or small models.
• Kernel/library coverage on older stacks or specific ops.
• Consistency with pretrained fp16 checkpoints or reproducibility requirements.

Otherwise bf16 without a scaler is the lower-maintenance choice.

Minimal verification steps for this case

1. Log scale per step on a short run:

   scaler = torch.amp.GradScaler('cuda')
   for step in range(N):
       with torch.amp.autocast('cuda', dtype=torch.float16):
           loss = model(...)
       scaler.scale(loss).backward()
       scaler.step(optimizer)
       scaler.update()
       print(scaler.get_scale())

2. Check for stabilization vs oscillation or collapse to 1. Constant step skipping indicates a problem.
3. Compare final loss/accuracy of fp16+scaler vs bf16 without scaler on a fixed seed and data subset to confirm parity for your model.
4. Confirm torch.__version__ and the amp docs for that release for constructor defaults and autocast usage.

One diagnostic detail that would change the recommendation: what is the observed scale trajectory and where infs/NaNs first appear — in forward, backward, or specific layers? If backoff is driven by a single op, fixing that op is preferable to retuning the scaler.