Reproducible HPC Stacks with Spack Environments

Problem: Software Stack Drift in HPC

On shared clusters, users often run ad‑hoc spack install commands that pick the newest compiler or the first available MPI provider. Over weeks, the stack on one node can differ from another by a compiler version, a variant flag, or even a dependency tree. This drift makes debugging reproducible results difficult and can silently break user code.

Thesis: Declare the Entire Stack in One Manifest

Spack environments let a team write a single spack.yaml that lists every package, compiler, and variant required for a project. When the environment is concretized, Spack resolves all abstract specs into a deterministic set of concrete package hashes. The resulting spack.lock can be committed to source control, ensuring that every developer, CI job, or production node uses the same stack.

What Is a Spack Environment?

An environment is a directory that contains:

• spack.yaml – the manifest of desired packages and configuration.
• spack.lock – the concrete resolution of the manifest.
• Optional .spackrc or spack.env files for environment‑specific configuration.

Typical workflow:

1. spack env create myenv – creates the directory.
2. Place spack.yaml inside myenv/.
3. spack env activate myenv – makes the environment visible to the shell.
4. spack concretize – resolves the manifest into a spack.lock.
5. spack install – builds or pulls the packages.
6. Use the stack via spack env activate or by loading the generated modules.

Concrete Example: A Tiny MPI Application

The following spack.yaml pulls in an MPI program, HDF5 with MPI support, and Python. All packages are built with GCC 10.2.0 and OpenMPI 4.1.5. A view named myview creates a merged filesystem prefix at /opt/myenv.

packages:
  all:
    compiler: [gcc@10.2.0]
    providers:
      mpi: [openmpi@4.1.5]

specs:
  - mpich@3.4.1
  - hdf5@1.12.0 +mpi
  - python@3.10.4

view:
  myview:
    prefix: /opt/myenv
    specs:
      - mpich@3.4.1
      - hdf5@1.12.0 +mpi
      - python@3.10.4

concretizer:
  unify: true

Steps to build and verify the stack on the first machine:

1. spack env create myenv
2. Copy the snippet above into myenv/spack.yaml
3. spack env activate myenv
4. spack concretize – Spack uses the Clingo ASP solver (the default) to resolve all constraints. A spack.lock file is written.

   Check: the file contains concrete package hashes and the exact compiler version.

5. spack install – packages are built or fetched from a build cache. If you have a build cache configured, many binaries will be pulled instead of compiled.

   Check: spack find -c lists the same concrete specs as in spack.lock.

6. ls /opt/myenv – the view prefix shows a merged filesystem containing the binaries and libraries of all packages.
7. spack containerize --dockerfile - > Dockerfile – generates a Dockerfile that reproduces the same stack in a container.

Verification on a Second Machine

To ensure reproducibility, clone the repository that contains spack.yaml and spack.lock. Then:

1. Check out the same Spack commit you used originally. For example, if you used the v0.18.0 tag, run git checkout v0.18.0 in the Spack checkout.
2. spack env activate myenv
3. spack concretize --check – this command compares the manifest to the lockfile and exits non‑zero if they differ.
4. spack find -c – the output should match the concrete specs in spack.lock.
5. Run docker build -t myenv . against the generated Dockerfile to confirm the container builds successfully.

Trade‑Offs and Limitations

• Concretization Time – For environments with dozens of packages, the ASP solver can take several minutes. This cost occurs only the first time you run spack concretize after changing the manifest.
• Spack Version Pinning – The same spack.yaml may resolve differently across Spack releases because default variant values or dependency rules evolve. The safest practice is to keep a copy of the Spack checkout you used (e.g., as a git submodule) and document its commit hash in the repository.
• Platform Sensitivity – spack.lock is platform‑specific. If you need the stack on a different OS or architecture, run spack concretize again on that target and commit a new lockfile.
• Binary Cache Trust – Using a build cache speeds up spack install, but you must trust the provider of the binaries. For maximum reproducibility, build from source on each machine or use a private cache you control.
• Bit‑identical Builds – The lockfile guarantees the same set of package hashes, but the compiled binaries may still differ due to compiler flags, environment variables, or subtle platform differences. Use the lockfile to pin the exact dependency graph; true bit‑identical builds would require identical build environments.

Best Practices for a Reproducible Workflow

• Keep spack.lock in version control alongside spack.yaml.
• Pin the Spack commit you use by checking out a specific tag or commit and documenting it (e.g., in a SPACK_COMMIT file). Do not rely on a runtime command to set the commit.
• Use spack concretize --check in CI to verify that the lockfile matches the manifest before building.
• Enable a view if you need a single prefix for modules or containers. Inspect the view with ls /opt/myenv.
• Generate a Dockerfile with spack containerize --dockerfile - and test the build locally before pushing to a registry.
• If you use a build cache, configure it in ~/.spack/config.yaml under config: build_cache and ensure all team members trust the same cache.

Actionable Takeaway

Eliminate software drift by creating a Spack environment, committing the spack.yaml and spack.lock, and using the same Spack checkout on every machine. The first run of spack concretize may take a few minutes, but thereafter every teammate and CI job will see the exact same compiler, package versions, and variants. Pin the Spack commit, keep the lockfile up to date, and prefer building from source or a trusted private cache to achieve the highest reproducibility. When a single prefix or container is needed, enable a view or run spack containerize and test the resulting image locally. The trade‑offs are mainly upfront concretization time and the need to manage Spack versions, but the payoff is a reproducible, auditable software stack that scales from a laptop to a supercomputer without surprises.