Spack Immutable Store: How Variant‑Rich Builds Coexist Safely

Requirements

In high‑performance computing (HPC) centers, a single software stack often needs to support many experimental builds of the same package – different compilers, MPI variants, debugging options, or performance flags. The core requirement is that each build be reproducible, isolated, and installable without overwriting or corrupting another build. Spack satisfies this by turning an abstract specification into a fully resolved dependency graph (the concretized spec) and mapping that graph to an immutable prefix in the store.

Design Overview

Spack’s store layout follows a content‑addressed scheme: the hash of the concretized spec determines the directory name under the store. The store is typically located at /opt/spack/opt/spack (or $SPACK_ROOT/opt/spack). Each directory contains a complete installation tree for that spec – compilers, variants, dependencies – and is never modified after creation.

Concrete Example

Consider two abstract specs that differ only by a debug variant:

spack concretize -f openmpi@4.1.1%gcc+debug
spack concretize -f openmpi@4.1.1%gcc~debug

Both commands produce distinct concretized specs. Running spack find shows two separate prefixes:

spack find | grep openmpi
openmpi@4.1.1%gcc+debug   1.0  /opt/spack/opt/spack/abcd1234
openmpi@4.1.1%gcc~debug    1.0  /opt/spack/opt/spack/efgh5678

The hash strings (abcd1234, efgh5678) are derived from the full concretized spec, ensuring uniqueness even when only a single variant differs.

Smallest Suitable Design

The minimal design that satisfies the requirements is:

• Fully resolve every spec before any installation step.
• Derive a deterministic, content‑addressed hash from the resolved spec.
• Create an immutable store prefix /opt/spack/opt/spack/<hash> that contains the entire dependency tree.
• Never modify a prefix once it exists; new builds use new prefixes.

This design guarantees that two builds of the same package with different variants never collide, and that the same spec will always map to the same prefix, enabling reproducibility.

Trust and Data Boundaries

Spack separates trust into two layers:

1. Core Engine: The Spack binary and its dependency‑resolution logic are considered trusted. It enforces the immutable store contract.
2. Package Recipes and Sources: Recipes are executable Python modules that define build logic. Sources are external archives that may include checksums. The engine verifies checksums but does not validate the recipe logic itself; changes to a recipe can alter concretization outcomes.

Binary caches and mirrors are additional trust boundaries: a cache may provide pre‑built binaries for a given hash. Verification of provenance is outside the core store layout guarantee.

Operational Checks

To ensure the design works as intended, perform the following checks:

• Concretization reproducibility: Run spack concretize -f twice on the same abstract spec and compare the concretized.yaml files. They should be identical.
• Store path existence: After installation, confirm that the prefix exists and contains the expected directories (e.g., bin, lib, include).
• Environment lockfile consistency: Create a Spack environment (spack env create myenv), add specs, run spack env lock, and later re‑concretize. The lockfile should enforce the same hash prefixes.

Failure Modes

Common failure scenarios include:

• Solver conflicts: The resolver may fail to find a consistent set of compiler and dependency versions. Inspect the solver log (spack concretize --log-file=solver.log) for details.
• Missing compilers or dependencies: If a required compiler is not available on the system, concretization will produce an error. Verify compiler installation with spack compiler find.
• Stale binary cache: A mirror may host outdated binaries that no longer match the hash. Use spack find --hash to verify that the binary cache contains the correct prefix.

When Design Changes

The immutable store model relies on the assumption that prefixes are never modified in‑place. If an environment requires:

• Mutable upgrades (e.g., patching a package after installation),
• Shared prefixes (multiple specs pointing to the same directory), or
• Non‑content‑addressed layouts (user‑defined prefix names),

then additional coordination is necessary. For example, a mutable upgrade would require a versioning scheme inside the prefix or a separate “overlay” directory to avoid overwriting the original build.

Practical Verification

To manually verify that two variant builds coexist:

1. Create a new environment:

   spack env create test-env
   cd test-env
   spack env activate .
   

2. Add both specs:

   spack add openmpi@4.1.1%gcc+debug
   spack add openmpi@4.1.1%gcc~debug
   

3. Lock the environment:

   spack env lock
   

4. Re‑concretize and confirm the lock file contains two distinct prefixes:

   spack env concretize
   cat lock.yaml | grep prefix
   

5. Install and check that both binaries are available in separate bin directories.

Conclusion

Spack’s hash‑based immutable store is a proven architecture for managing variant‑rich builds in HPC. By fully resolving specs before installation and mapping them to unique prefixes, it eliminates name collisions and guarantees reproducibility. Operational checks and awareness of failure modes help maintain the integrity of the store, while understanding the trust boundaries ensures that only verified sources contribute to the final binaries.