Slow flake rebuilds: diagnosing whether NixOS evaluation is the real bottleneck
0 reputation · 02 Apr 2021, 23:09 UTC
0 reputation · 02 Apr 2021, 23:09 UTC
A flake-based NixOS configuration that pulls in a large module set can spend a long stretch on nixos-rebuild before any derivation starts building. Nix evaluation is single-threaded, the entire configuration is re-evaluated on every rebuild, and wall-clock time blends evaluation, input fetching, and build planning into one number.
The goal is to measure before optimizing: confirm whether evaluation is actually the dominant phase before restructuring imports, trimming flake inputs, or reaching for parallel evaluation tooling. On a recent stable release (NixOS 25.05 with Nix 2.24 or newer, for example), the evaluator exposes statistics and tracing instrumentation, but flag names and output formats have shifted across versions, so the supported method on current Nix needs confirmation.
Two constraints complicate attribution: locked inputs still trigger full re-evaluation of the module system, and hardware differences such as single-core speed and store size can dominate comparisons between machines.
31775 reputation · 03 Apr 2021, 07:03 UTC
Evaluation is a plausible bottleneck because Nix evaluates the entire module system single-threaded on every rebuild, even with locked inputs. But you must measure it — don't assume. The reliable way on current Nix (2.18+/2.24+/Lix) is to time a pure evaluation of the toplevel derivation path and compare it against the full nixos-rebuild wall-clock time.
Measure pure evaluation time (no fetch, no build planning):
time nix eval --raw .#nixosConfigurations.<host>.config.system.build.toplevel.drvPathThis forces the evaluator to produce the toplevel derivation path. It exercises the module system, option merging, and derivation construction — the bulk of evaluation work.
Measure the full rebuild dry-run (includes fetch if inputs missing, plus instantiation and build planning):
time nixos-rebuild dry-build --flake .#<host>Isolate fetch time by repeating both commands with a warm store and locked inputs, then adding --offline to the rebuild:
time nixos-rebuild dry-build --flake .#<host> --offlineThe difference between the two time runs approximates fetch overhead. What remains in the rebuild after subtracting the nix eval time is instantiation and build planning.
Flag names and stats output have shifted across Nix 2.18, 2.24, and Lix. Do not trust older blog posts. Verify on your machine:
nix --version
nix show-config | grep -i eval
man nix.conf # check the installed manual for eval-cache and stats optionsCommonly available on recent releases:
NIX_SHOW_STATS=1 — prints aggregate evaluator counters (values allocated, thunks, function calls, GC time, CPU time) to stderr.NIX_COUNT_CALLS=1 — counts function calls (heavy output, use sparingly).--option eval-cache true — enables the evaluation cache (persisted across invocations when the store allows).These give aggregate evaluation effort, not a per-module breakdown. They also don't map cleanly to wall-clock time because GC pauses and memory pressure distort the relationship.
Not available as a supported counter. Nix exposes only aggregate evaluator statistics. Fine-grained attribution requires either:
If you need per-module costs, bisection is the only practical path today.
Heuristic: if measured evaluation time is below roughly 20–30% of total rebuild wall-clock time, restructuring imports and trimming modules is unlikely to pay off relative to the maintenance cost. Target fetch parallelism, substituter configuration, or hardware (single-core speed, store on SSD) instead.
nix eval of the toplevel drvPath excludes some work nixos-rebuild performs (activation script evaluation, some module finalization). Treat it as a close approximation, not exact./nix/store sizes slow stat-heavy operations. Cross-machine comparisons need normalized conditions.eval-cache option may be named differently or absent — confirm against the installed manual.What is the output of nix --version on the machine where you're seeing slow rebuilds? If you're on Nix 2.18 or an older Lix release, the stats output format and eval-cache behavior differ, and the measurement approach may need adjustment.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.