Spack Concretization: Turning Flexible Specs into Reproducible Builds

The Problem: Same Command, Different Build

You run spack install hdf5 +mpi and get a working library. A colleague runs the same command a week later and gets a different build. Nothing is broken, but you cannot reproduce each other's environments. The root cause is that Spack does not install what you typed; it installs what the concretizer decided you meant.

Concretization is the process that turns an abstract spec—your flexible request with version ranges, variant flags, and dependency preferences—into a concrete directed acyclic graph (DAG). Every node in that graph is a fully resolved package with a specific version, compiler, and variant set, identified by a unique hash. This step is what makes Spack reproducible, and it is also where most confusion about changed builds begins.

What Concretization Actually Resolves

An abstract spec is intentionally loose. You might ask for hdf5 +mpi without pinning the MPI implementation, the compiler version, or the HDF5 release. Concretization closes those gaps by consulting the package repository, applying your constraints, and selecting a consistent set of concrete packages.

The solver considers several dimensions at once:

• Versions: It picks a specific release that satisfies any range you provided and the requirements of dependent packages.
• Variants: Boolean and multi-valued options like +mpi, ~fortran, or cxxstd=17 are resolved to fixed values.
• Virtual providers: When a package depends on a virtual like mpi, the concretizer selects a concrete provider such as openmpi or mpich based on your preferences, package defaults, and any policies configured in packages.yaml.

The output is not a flat list. It is a DAG where each edge represents a dependency relationship, and each node carries a hash. That hash is the identity Spack uses to decide whether a package is already built, whether it exists in the build cache, and whether two environments are truly identical.

A Worked Example with an Environment

The most reliable way to see concretization at work is inside a Spack environment, because environments persist the result to a lock file.

Run these commands in your shell. No special permissions are required beyond the ability to write to your Spack installation and environment directory.

# Create and activate an environment
spack env create repro-test
spack env activate repro-test

# Add an abstract spec with a dependency constraint
spack add hdf5 +mpi ^openmpi %gcc@11.2.0

# Force concretization and inspect the result
spack concretize -f

The -f flag forces re-concretization even if a lock file already exists. After it completes, Spack writes a spack.lock file inside the environment directory. This file records the concrete DAG, including every hash.

To verify that the environment matches the lock file, run:

spack find --format '{hash} {name}@{version}'

You should see a list where every package has a hash and a pinned version. If you copy the environment directory to another machine with the same Spack version and run spack install, Spack should build or fetch exactly those hashes.

One risk to note: if you edit packages.yaml or the package repository between concretization and installation, the next concretization may produce a different DAG even if your spec did not change.

The Reproducibility Trade-off

Concretization is powerful, but it is not free. For large dependency trees—common in HPC stacks with MPI, BLAS, and GPU support—the solver can explore many version and variant combinations before settling on a solution. This exploration is computationally expensive and can take minutes on complex environments.

There is also a versioning hazard. Concretization results depend on the package repository and the concretization policies bundled with your Spack release. If you upgrade Spack, or if the package repository adds new versions or deprecates variants, the same abstract spec may concretize differently. The hash you trusted last month may no longer be the one you get today.

Build caches add another layer of risk. Spack reuses cached builds by hash, which is efficient, but if you switch compilers or architectures without clearing or segregating the cache, you may pull a binary built for an incompatible target. One diagnostic is to compare the concrete spec before and after clearing the build cache to confirm determinism, but be aware that spack buildcache clear removes cached artifacts. Any packages not present locally will need to be rebuilt or re-downloaded, so treat it as a state-changing diagnostic, not a routine check.

A safer verification is to snapshot the hashes from spack find in one environment, create a second environment with the same abstract spec on the same Spack version, and compare the resulting lock files. If the hashes match, your concretization is deterministic for that repository state.

Practical Takeaways

Treat concretization as the contract between your request and your build. Do not rely on an abstract spec alone for reproducibility; commit the spack.lock file to version control and share it with collaborators. When debugging a changed build, compare hashes rather than package names—two environments can both contain hdf5 and still be different if the hashes diverge.

If you need to confirm that a cache or repository change altered your DAG, capture the hash list before and after the change using spack find --format '{hash} {name}@{version}'. If the lists differ, the concretizer found a different solution, and your build is no longer bit-for-bit reproducible with the previous environment.

Finally, remember that concretization behavior can evolve between Spack releases. Pin your Spack version in CI and document the version used to generate any lock file you distribute. That single habit prevents most of the "it worked on my machine" problems that concretization is designed to solve.