When Your HPC Stack Drifts: Using Spack Lockfiles for Reproducible Builds

The problem: silent stack drift

You build a scientific software stack on Monday. It compiles, the tests pass, and the paper gets written. Six months later a colleague tries to reproduce the result on a fresh cluster node and gets a segfault in a different MPI library. Nothing in the source code changed, but the stack underneath it did.

This is the concretization gap. Spack specs like hdf5 +mpi are abstract requests. The concretizer—the solver that turns those requests into concrete build recipes—picks the newest compatible compiler, MPI implementation, and dependency versions available at the moment you run it. If you re-run the solver later, you can get a different answer.

Environments and lockfiles: freezing the graph

A Spack environment is a directory containing a spack.yaml file. It defines a collection of specs as a single unit. When you install into that environment, Spack writes a spack.lock file beside it. The lockfile is the concrete output: exact versions, compiler identities, variant settings, and content hashes for every package in the dependency graph.

The practical distinction is between letting the solver choose fresh each time versus trusting a previously solved graph. In day-to-day development, you often want the solver to pick up new compatible releases. For a paper, a production deployment, or a shared container, you want the graph frozen. The lockfile is that frozen state. If you change a spec in spack.yaml and want to update the lockfile without building yet, run spack concretize inside the environment.

Worked example: HDF5 and NetCDF-C

Suppose you need an HDF5 build with MPI support and a NetCDF-C library linked against it, all built with a specific compiler toolchain. Create a fresh directory and initialize an environment:

spack env create -d . myenv

Then edit the generated spack.yaml to pin the toolchain. The versions below are illustrative; substitute the compiler and MPI release available on your system:

spack:
  specs:
  - hdf5 +mpi
  - netcdf-c +mpi
  packages:
    all:
      compiler: [gcc@12.3.0]
      providers:
        mpi: [openmpi@4.1.5]

Activate the environment and install:

spack env activate -d ./myenv
spack install

Spack concretizes the specs, builds the packages, and writes spack.lock. Open that file and you will see a JSON document with a concrete section for every package, including the full hash and the exact dependency graph. To confirm the environment is active and only the locked packages are visible:

spack find --loaded

If you hand this directory to a colleague with the same Spack version and the same config.yaml, packages.yaml, and compilers.yaml, the install should reuse the locked hashes rather than re-solving.

Trade-offs and limitations

Lockfiles are not free. Maintaining them adds overhead when upstream packages change frequently. If a security fix lands in OpenMPI next week, your locked stack will not pick it up until you deliberately re-concretize. Teams need a policy: regenerate lockfiles on a schedule, or regenerate only when a specific upstream change matters.

Two coupling risks matter more. First, lockfiles are tied to the Spack version that generated them. A significantly older or newer Spack may fail to concretize because package recipes or solver logic changed. Second, environment isolation depends on host configuration. If your packages.yaml prefers a different MPI provider or your compilers.yaml lists a different GCC path, the same lockfile can produce mismatched hashes on a different machine.

What to do next

If you are about to publish a stack or hand it to a collaborator, commit both spack.yaml and spack.lock to version control. Record the Spack version used to generate the lockfile—ideally by pinning Spack itself in a container or module. Before sharing, verify on a clean node: create the environment, run spack install, and check that spack find --loaded matches the locked package list. Reproducibility is not a property of the lockfile alone; it is a property of the lockfile plus a controlled Spack version plus a controlled host configuration.