Limits of Spack Build Cache Reuse When Compiler Version Changes

Question

Spack’s build cache stores pre‑built packages using a key that hashes the full spec, target architecture, and the complete compiler string (version and flags). The goal is to evaluate whether the cache can safely treat patch‑level changes in the compiler version as compatible, thereby increasing hit rates without violating ABI guarantees.

Currently the cache invalidates on any change to the compiler string, and the policy for what constitutes a ‘compatible’ compiler remains an open discussion in the community. This conservatism avoids silent mis‑links but may lead to unnecessary rebuilds when only minor compiler updates are applied. Should Spack relax the compiler hash to ignore patch‑level revisions? What objective criteria could define a compatible compiler version? How would adjusting the hash affect cache hit rates and the reproducibility of builds?

Tasadduq Burney · Answer

Answer to the Question
Spack’s build‑cache key is built from the full compiler spec string (e.g., gcc@11.3.0 or gcc@11.3.1). Because the string is hashed verbatim, any change to the patch level produces a new key and forces a rebuild. The policy is intentionally conservative to preserve bit‑for‑bit reproducibility and avoid silent ABI mismatches.
Confirmed Facts

Cache keys include the exact compiler version; patch releases are not considered interchangeable.
Spack has no built‑in notion of a “compatible compiler” equivalence class.
Patch releases can modify ABI‑affecting behavior (e.g., GCC 11.3.1 fixed a libstdc++ ABI tag regression, Clang patch releases changed template mangling).
Reproducibility guarantees (e.g., spack spec --cache-only, binary mirrors) rely on the full spec hash.

Likely Explanation for the Current Conservatism
Spack’s designers chose a strict hash to avoid the risk of linking against binaries that were built with different code‑generation or library assumptions. A patch‑level change can alter exception handling, inlining decisions, or even the layout of standard library types. Treating such changes as equivalent could silently break user programs.
Possible Objective Criteria for “Compatible” Compilers

Per‑compiler ABI stability policies (e.g., GCC’s claim of ABI compatibility within a major version for libstdc++).
Explicit version ranges that exclude known ABI‑breaking patches.
A maintained mapping table that lists which patch revisions are safe to treat as the same cache namespace.

Impact of Relaxing the Hash
Removing the patch component would increase hit rates only if the same patch exists in the mirror, which is uncommon. More importantly, it would break reproducibility guarantees: two users on slightly different patch levels would see identical binaries, but those binaries might not be interchangeable.
Practical Steps for Your Situation

Verify the cache key difference:
spack spec --cache-only package@1.2.3 %gcc@11.3.0
spack spec --cache-only package@1.2.3 %gcc@11.3.1

Consult the release notes of the compiler patch to check for ABI‑affecting changes.
Run a minimal binary compatibility test (e.g., build a small program with each compiler, then link against a library built with the other).
If you must share a cache across patch levels, consider a local wrapper that normalizes the spec string, but be aware that this is not supported by upstream binary mirrors.

Missing Diagnostic Detail
Do you mainly want to reduce rebuild frequency, or have you observed runtime failures after a compiler update? Knowing that will help decide whether a local key normalizer is appropriate or if you should keep the conservative policy.

Limits of Spack Build Cache Reuse When Compiler Version Changes

1 answer

Answer to the Question

Confirmed Facts

Likely Explanation for the Current Conservatism

Possible Objective Criteria for “Compatible” Compilers

Impact of Relaxing the Hash

Practical Steps for Your Situation

Missing Diagnostic Detail

Your perspective belongs here.

0 question comments