PyTorch torch.compile Training Speed‑Up Guide

Problem: Eager execution stalls training speed

When you train a deep model in PyTorch’s default eager mode, each operation launches its own CUDA kernel. This separation adds overhead, especially for models with many small layers, and can make each epoch feel slower than necessary.

Takeaway: wrapping the model with torch.compile tells PyTorch to capture a static graph, fuse operators and pick efficient kernels, often cutting epoch time by a noticeable margin with only a one‑line change.

How torch.compile works under the hood

PyTorch 2.0+ includes a just‑in‑time compiler that, on the first call, traces the model’s forward pass into a graph. The compiler then applies rewrites such as operator fusion, layout optimization and selects kernels from back‑ends like CUDA, CPU or community‑provided Indie backends. The resulting artifact is cached, so subsequent iterations reuse the compiled code and avoid the per‑operation launch overhead.

Worked example: compiling a ResNet‑50 training loop

import torch, torchvision, time

# Assume PyTorch >= 2.0
model = torchvision.models.resnet50()
model = model.to('cuda')
# Wrap the model – this triggers compilation on the first forward pass
model = torch.compile(model, mode='reduce-overhead')

optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
criterion = torch.nn.CrossEntropyLoss()
# Dummy data loader – replace with your real dataset
dummy_input = torch.randn(32, 3, 224, 224, device='cuda')
dummy_target = torch.randint(0, 1000, (32,), device='cuda')

# Warm‑up run to pay the compilation cost
model(dummy_input)
optimizer.zero_grad()
loss = criterion(model(dummy_input), dummy_target)
loss.backward()
optimizer.step()

# Timed epoch (repeat a few times for a stable average)
epoch_times = []
for _ in range(5):
    start = time.time()
    optimizer.zero_grad()
    loss = criterion(model(dummy_input), dummy_target)
    loss.backward()
    optimizer.step()
    epoch_times.append(time.time() - start)

print(f'Average epoch time: {sum(epoch_times)/len(epoch_times):.3f} s')

Run the script on a machine with a CUDA‑capable GPU (e.g., a V100). The first iteration includes the compilation latency; after the warm‑up you should see a lower average epoch time compared to the same script without the torch.compile wrapper. To confirm that compilation succeeded, add:

print(torch.compiler.is_enabled())  # should print True

If you want to see what the compiler did, enable verbose mode:

model = torch.compile(model, mode='reduce-overhead', verbose=True)

This prints the captured graph and shows which operators were fused.

Trade‑offs and limitations

• Startup cost. The first forward pass after wrapping can take seconds to minutes as the graph is traced and kernels are selected. Always benchmark after a warm‑up iteration.
• Dynamic control flow. Models that rely on Python‑level conditionals or loops that depend on tensor values may not be traceable. In such cases torch.compile will raise an error; you can either rewrite the problematic section or disable compilation for that module.
• Custom ops and third‑party layers. A custom torch.autograd.Function or a layer from an external library that uses unsupported operations will cause compilation to fall back or fail.
• Numerical differences. Because graph execution can reorder floating‑point operations, results may differ slightly from eager mode. If exact reproducibility is required, enable PyTorch’s deterministic flags or compare outputs within a tolerance.

Actionable closing

Start with the low‑overhead mode:

model = torch.compile(model, mode='reduce-overhead')

Measure epoch time after a warm‑up run. If you observe a speed‑up, keep the wrapper. If you encounter errors, try:

torch.compiler.set_strict_mode(False) to relax tracing constraints,
Inspect the console output for warnings about unsupported ops,
Fall back to eager mode for the problematic module by wrapping it with torch.compile(module, disable=True) (available in recent PyTorch releases).

With these steps you can decide whether torch.compile delivers a worthwhile training speed‑up for your specific model and hardware.