Question
Inconsistent bilinear interpolation results when align_corners=None across CPU and GPU backends
Rahul Vale
0 reputation · 09 Aug 2022, 18:39 UTC
112.3K views1
Loading…
0 reputation · 09 Aug 2022, 18:39 UTC
Goal: obtain identical interpolation outputs from torch.nn.functional.interpolate with mode='bilinear' and the default align_corners=None when running the same model on CPU and CUDA devices.
Constraint: the default value leaves the interpolation kernel unspecified, causing the CPU and CUDA backends to use different resampling schemes, which produces non‑identical results and harms reproducibility. The documentation notes that users should set align_corners to True or False for deterministic behavior, but the default remains None in all released releases, leaving an open decision about whether a future version will change the default or unify the kernels.
Will a future PyTorch release change the default align_corners value to True or False? Should users always specify align_corners to ensure device‑agnostic reproducibility? Is there a plan to make the CPU and CUDA implementations consistent when align_corners=None?
35275 reputation · 10 Aug 2022, 02:36 UTC
No, do not rely on align_corners=None for device-agnostic reproducibility. Specify align_corners=True or False explicitly in torch.nn.functional.interpolate and grid_sample. There is no plan to make CPU and CUDA consistent while align_corners=None is used; the sentinel is deprecated and intentionally non-portable. Whether a future PyTorch release will change the default value from None to a concrete boolean is not settled, and the default remains None in released versions to date. Changing the default would be a breaking semantic change, so the project has moved toward deprecation and removal of the sentinel rather than unifying kernels under it.
pytorchalign_corners=None is a deprecated legacy sentinel for interpolate and grid_sample. It is not a stable setting.None.align_corners value, small numerical differences can remain due to float32 reduction order and kernel precision. Systematic divergence indicates different mapping semantics, not just rounding.With align_corners=None the library maps the sentinel to different effective semantics on CPU vs CUDA in certain PyTorch versions. The legacy coordinate normalization differs between the True-like and False-like paths, so you see systematic output differences rather than only low-magnitude noise.
align_corners=None with an explicit boolean that matches your intended grid. Use the same explicit value on CPU and CUDA.import torch
import torch.nn.functional as F
x = torch.arange(16, dtype=torch.float32).view(1,1,4,4)
def run(dev):
t = x.to(dev)
# explicit value, e.g. False
y = F.interpolate(t, size=(8,8), mode='bilinear', align_corners=False)
return y.cpu()
cpu_out = run('cpu')
gpu_out = run('cuda') if torch.cuda.is_available() else None
With an explicit align_corners, systematic divergence should disappear, leaving only expected low-magnitude numerical noise. Compare with a tolerance, not exact equality, e.g. torch.allclose(cpu_out, gpu_out, atol=1e-5, rtol=1e-5).
Note: changing from None to an explicit boolean is a semantic change to the resampling grid. Validate downstream metrics after the change.
One missing diagnostic detail that changes the recommendation: the exact PyTorch version and the full call signature you use, including size vs scale_factor, mode, and backend. The legacy mapping for None is version sensitive, and whether to pin to True or False depends on which legacy behavior you want to preserve.
Use comments to ask for clarification. Post a solution as an answer.
5,960 reputation · 09 Aug 2022, 21:29 UTC
To quantify residual differences after pinning align_corners, run the identical F.interpolate call on CPU and CUDA tensors and compare with torch.allclose(atol=1e-5, rtol=1e-5). Document the maximum absolute deviation and whether it falls within your application's tolerance budget. This empirical check is necessary because even with an explicit boolean, floating-point reduction order and memory format (contiguous vs. channels-last) can produce sub-ULP differences that allclose will catch.
Since PyTorch 1.10, antialias=True is supported for bilinear downsampling. When enabled, the CPU and CUDA paths converge more closely because both use the same pre-filtering logic, but the flag must be set explicitly—its default is False for backward compatibility. If your pipeline involves downsampling, adding antialias=True alongside a fixed align_corners value further reduces cross-device drift.
Record the exact PyTorch and cuDNN versions in your reproducibility manifest. Minor releases have adjusted the coordinate mapping for align_corners=False (e.g., 1.13 vs. 2.0), so a version upgrade can shift results even when all flags are explicit.