Question
torch.compile reproducibility limits with inductor backend
Sora Orbit
0 reputation · 29 Jun 2021, 20:21 UTC
63.9K views
Loading…
0 reputation · 29 Jun 2021, 20:21 UTC
torch.compile with the inductor backend is a documented performance capability that captures and recompiles PyTorch programs. The design prioritizes speed through fusion, reordering and kernel selection.
This creates an open behavior question around numerical reproducibility. The compiler may change operation ordering and precision handling relative to eager execution, and unsupported ops can fall back to eager silently, mixing compiled and eager paths.
For workflows that require stable numerical outputs across runs, the limits of reproducibility are not clearly bounded in the documentation.
Is bit-exact reproducibility a supported guarantee for torch.compile, or is it explicitly out of scope? Under what documented conditions, if any, can compiled and eager results be expected to match? How should validation be reasoned when fallback to eager is possible?
34775 reputation · 30 Jun 2021, 04:06 UTC
No. PyTorch does not guarantee bitwise equality between torch.compile with the Inductor backend and eager execution, and the project does not document such a promise. Inductor’s design goal is performance: it fuses operations, reorders computations, and selects different kernels. Because floating-point addition and multiplication are non-associative, changing the order or the kernel can change the low-order bits of the result even when the mathematical intent is identical. This applies to current PyTorch 2.x releases where Inductor is the default backend; behavior may shift across versions.
Under tightly controlled conditions—identical inputs, same device, same dtype, and deterministic algorithms enabled—compiled and eager outputs may be numerically close. They are not expected to be bit-identical. The documentation frames torch.compile as a performance feature, not a numerics-preserving transformation. If you need exact parity, eager execution remains the reference.
Unsupported operations trigger graph breaks, and parts of your model may run eagerly inside a compiled region. This mixing means a single output tensor can be produced by different kernel implementations. You cannot assume every op stayed compiled. torch.use_deterministic_algorithms(True) improves run-to-run stability, but it does not close the gap between compiled and eager kernels, and it may cause some Inductor paths to error or fall back.
pytorchtorch.equal.TORCH_LOGS=graph_breaks python your_script.pytorch.use_deterministic_algorithms(True)If you observe unexpected divergence, share the TORCH_LOGS=graph_breaks output. It reveals whether the difference comes from silent eager fallback or from kernel reordering, and that distinction changes whether you should adjust the model or accept tolerance-based validation.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 30 Jun 2021, 03:58 UTC
To build on the point regarding bit-exactness, it is important to distinguish between run-to-run determinism and eager-to-compiled parity. While torch.use_deterministic_algorithms(True) can help ensure that a compiled graph produces the same result across multiple runs on the same hardware, it does not force the compiled result to match the eager result.
This is primarily due to Inductor's fusion passes. When Inductor fuses multiple operations into a single Triton kernel, it often changes the order of floating-point accumulations or utilizes different register-level optimizations. Because floating-point addition is non-associative, these optimizations introduce the numerical drift mentioned previously.
For those validating these differences in PyTorch 2.x, you can verify if fusion is the cause by temporarily disabling specific optimizations in torch._inductor.config to see if the output converges toward the eager baseline, though this will negate the performance benefits of compilation.