Limits of stdlib @time versus BenchmarkTools.jl for micro‑benchmarks

Reliable Steady-State Timing

For reliable steady-state timings in any Julia 1.x release, the recommended combination is BenchmarkTools.jl using @btime or @benchmark. While @time is available in the standard library, it measures a single execution. Because Julia uses Just-In-Time (JIT) compilation, the first call to a function includes the time required to compile the code, which heavily skews the result.

BenchmarkTools.jl solves this by performing a "warm-up" phase, executing the code multiple times to ensure compilation is complete before recording samples. It then provides a median execution time and a distribution, which filters out noise from background OS processes and garbage collection spikes.

Interpreting Compile-Time Percentages

In recent Julia 1.x releases, @time output includes a percentage indicating how much of the total wall-clock time was spent on compilation. This should be interpreted as a diagnostic flag rather than a performance metric:

• High Percentage: The measurement is dominated by JIT overhead and does not reflect the actual execution speed of the compiled machine code.
• Low/Zero Percentage: The function was likely already compiled (warm), and the timing is closer to the steady-state performance.

When comparing across Julia versions, these percentages are not directly comparable. Improvements in the JIT compiler or changes in inlining behavior between point releases (e.g., 1.10 to 1.11) mean that a 10% compilation overhead in one version may represent a different amount of actual work or a different optimization path than in another.

Justifying the BenchmarkTools Dependency

The added dependency of BenchmarkTools.jl is justified when you move from coarse profiling to micro-benchmarking. Use the standard library @time only for high-level checks where execution takes seconds or minutes, making JIT overhead negligible. Switch to BenchmarkTools.jl when:

• The function executes in milliseconds or microseconds.
• You are comparing two different implementation strategies to find a bottleneck.
• You need to verify that a change actually reduced allocations (using @benchmark to see the full memory profile).

Verification Workflow

To verify the difference in your current environment, run the following sequence:

# First run: includes JIT overhead
@time my_function()

# Second run: reflects warmed-up state
@time my_function()

# Using BenchmarkTools for statistical reliability
using BenchmarkTools
@btime my_function()

Diagnostic Detail: Are you benchmarking functions that call external C or Fortran libraries? If so, note that neither @time nor @benchmark tracks memory allocations occurring inside those external libraries.