BenchmarkTools.jl and Profile.jl: Distinguishing Compilation Overhead from Runtime Bottlenecks

The short answer

Profile.jl samples whatever runs, including the JIT compiler. The recommended pattern is to execute the function before profiling, with the same argument types and code paths production will hit, then clear the profiler state and sample only warmed executions. @benchmark already does this internally, which is why its numbers and a naive profile often disagree.

Recommended warm-up pattern

1. Call the function once or twice with production-representative arguments (same types, shapes, and branches).
2. Let GC settle — optionally run GC.gc() so warm-up allocation spikes don't pollute samples.
3. Reset the profiler, then sample the real workload.

using Profile

# 1. Warm up: trigger compilation of all production paths
myfunc(warmup_data)
GC.gc()

# 2. Clear any stale samples and start fresh
Profile.clear()

# 3. Profile only warmed executions
@profile for _ in 1:1000
    myfunc(production_data)
end

Profile.print()

Key detail: the loop inside @profile matters. A single warmed call is often too short for the sampler (default interval ~1 ms) to collect meaningful counts; running many iterations gives the sampler enough ticks to rank functions reliably.

Why warm-up must match production

Julia specializes compiled code per concrete input types. If you warm up with Vector{Float64} but profile with Vector{Float32}, the profiler will capture a fresh compilation pass. The same applies to branches: if a dispatch path is only triggered by certain data, warm-up must trigger it too. Otherwise your profile mixes compiler frames (look for jl_compile, typeinf, or Core.Compiler frames in the output) with runtime logic, and the hotspots are meaningless.

Cross-validating Profile hotspots against @benchmark

The two tools answer different questions: @benchmark gives you a statistical distribution of total runtime; Profile tells you where that time goes. To confirm they agree:

1. Run @benchmark myfunc($data) and note the median and the spread (e.g., interquartile range or the min/max gap). A tight distribution means a stable workload worth profiling.
2. Profile a warmed workload of comparable total duration.
3. Compare: the functions with the highest cumulative sample counts in Profile.print() should be the same functions that dominate the benchmarked time. If the benchmark shows high variance but the profile shows a single dominant hotspot, the variance likely comes from GC or allocation, not the hotspot itself — check allocation with @benchmark's memory estimate or @allocated.

Caveats

• Profile is sampling-based: sample counts approximate time and are noisy for short runs. Don't trust rankings built on a handful of samples.
• Compilation can reappear mid-profile if new types or branches are encountered after warm-up.
• Warming with atypical inputs creates specialization bias — the compiled code you profile may not be what production runs.

This behavior is well-established across Julia 1.x, but sampler defaults and compiler frame names can shift between versions, so verify on your installed version with a trivial function before trusting the workflow on production code.