torch.profiler schedule produces overlapping traces and inflated kernel times — how to configure it correctly

I'm using torch.profiler with the Kineto backend (PyTorch 2.x) to find GPU bottlenecks in a training loop before attempting any optimization. The goal is a clean, steady-state trace of a few representative iterations that I can inspect in TensorBoard, without the profiler itself distorting the numbers I'm trying to measure.

Two things are unclear. First, the schedule(wait=..., warmup=..., active=..., repeat=...) semantics: it's not obvious how profiler.step() interacts with my loop boundaries, and whether traces from consecutive repeat cycles can overlap or overwrite each other when using on_trace_ready with tensorboard_trace_handler. Second, overhead: for small kernels the profiler appears to inflate durations substantially, and I'm unsure which knobs (record_shapes, with_stack, profile_memory) are safe to enable without skewing GPU timing attribution.

My questions:

• What is the correct placement of profiler.step() relative to the training iteration so that wait/warmup/active counts map to whole iterations?
• Which profiler options meaningfully reduce measurement overhead while still giving operator-level self CUDA time?
• How can I quantify the profiler's overhead itself (e.g., against torch.cuda.Event timing) to know whether a measured hotspot is real?