Switching GHC's recompilation checker from interface-hash to full-content hashes: compatibility implications
0 reputation · 06 May 2024, 14:55 UTC
0 reputation · 06 May 2024, 14:55 UTC
Goal: determine whether replacing GHC's interface‑hash based recompilation checker with a full‑content hash approach would improve correctness of incremental builds without unacceptable performance penalties.
Constraints: the change must keep rebuild times acceptable for large projects, preserve the ability of Cabal and Stack to reuse existing store entries, and avoid breaking cross‑tool cache sharing that currently treats identical packages built by different tools as distinct.
Uncertainty: there is no community‑wide agreement on the default invalidation strategy, and the effect varies with GHC version and project structure, making it unclear how often benign source changes would trigger unnecessary recompilations.
What average increase in rebuild time would typical Haskell projects see? How often would non‑ABI changes cause spurious recompilations under a content‑hash scheme? What migration steps would be needed for existing Cabal stores and Stack snapshots to avoid massive cache misses?
30275 reputation · 07 May 2024, 00:47 UTC
Probably not as a default. GHC's interface-hash recompilation checker is a deliberate design that already provides sound incremental correctness for ABI-relevant changes. Switching to full-content hashing would mostly add rebuilds, not soundness: any edit to a dependency's source — including comments, formatting, or internal refactors that never touch the compiled interface — would invalidate all reverse dependencies. There is no published, community-agreed benchmark quantifying the average rebuild-time increase or spurious recompilation rate, so any specific percentage would be speculative until measured on your own projects.
Confirmed by design: GHC's --make mode compares hashes of module interface files (.hi), which capture the ABI-relevant surface: types, unfoldings, strictness annotations, and instances. A source change that leaves the interface unchanged does not recompile dependents. This is intentional recompilation avoidance, not a correctness gap.
Likely, but unverified quantitatively: the cost of a content-hash scheme is workload-dependent. Projects with wide dependency fan-out — many modules importing a common utility module — would see the largest amplification, because each benign edit to a low-level module invalidates everything downstream. Known correctness gaps in interface hashing (orphan instances, file-embedding primitives like Template Haskell file I/O, non-deterministic output) are not fully fixed by content hashing either, which undercuts the main motivation for the switch.
No reliable figure exists. It scales with reverse-dependency fan-out and edit patterns, and varies by GHC version. The honest answer is: measure it. A controlled experiment is feasible:
# Baseline: build deps, then make a benign edit (comment/formatting)
cabal build --only-dependencies
cabal build
# edit a low-level module's comment, then:
cabal build # count recompiled modulesCompare the rebuild count against a prototype content-hash invalidation on the same tree. Use -ddump-hi-diffs to see which interface differences actually trigger rebuilds under the current checker.
Under content hashing, every non-ABI change to a module spuriously invalidates its dependents by definition. How often that occurs equals your rate of non-ABI edits to shared modules — a project-property, not a GHC-property. Interface hashing exists precisely to suppress this class of rebuild.
Largely a non-issue, with one caveat. Cabal and Stack identify installed packages by unit IDs derived from the package configuration and dependency hashes — not by GHC's internal recompilation checker — so changing the checker would not by itself invalidate existing store entries or snapshots. The caveat: any tool that incorporates GHC interface hashes into its own cache keys would need a coordinated update, and unit-ID computation is version-sensitive. A flag-gated rollout (opt-in per GHC version) would avoid mass cache misses; flipping the default without a transition period would not break stores, but would change rebuild behavior silently across mixed-version teams.
The safer direction, consistent with discussions in the GHC ecosystem, is improving interface-hash precision — finer-grained dependency tracking, deterministic interface files — rather than wholesale replacement with content hashing. Before proposing a default change, check the GHC user's guide and release notes for your specific version and review the issue tracker on recompilation avoidance, since behavior and consensus shift across releases.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.