Spack Concretization and Lockfiles: Architecture Note for Reproducible Environments

Problem: deterministic dependency graph from abstract intent

Teams need a repeatable way to turn an abstract Spack spec such as mpileaks ^openmpi@4.1: into a fully specified dependency DAG with exact versions, compilers, variants and hashes. Concretization is that step, and the lockfile spack.lock is the artifact that records the result so another machine can reproduce the same graph without re-resolving.

The useful takeaway is that concretization is deterministic only within a fixed configuration closure. Change scopes, compilers, or the package repository and the lockfile is no longer valid.

Requirements for concretization

Minimum inputs are the abstract spec and the current package repository. The resolver also needs the active configuration scopes: packages.yaml, compilers.yaml, config.yaml. These scopes supply version constraints, compiler defaults, variant preferences and mirror settings.

The output is a concrete spec for each node in the DAG and a lockfile that stores those concrete specs in a deterministic order. The lockfile is intended to be checked into source control for an environment.

Smallest suitable design

The minimal design keeps resolution self-contained. The concretizer reads the abstract spec, loads package metadata from the repository, applies constraints from configuration scopes in priority order, and runs an internal constraint solver to select versions, compilers and dependencies.

No network access is required for pure source concretization. Network is only consulted when binary caches or mirrors are enabled for fetching prebuilt packages.

Example workflow on a developer workstation with Spack installed for the current user:

spack env create -d myenv
spack env activate myenv
spack add mpileaks ^openmpi@4.1:
spack concretize -f
spack lock --include-dependencies

Run as an unprivileged user. The commands mutate the environment directory and write spack.lock. Risk: higher-priority configuration scopes can silently override user intent because scope order is evaluated top-down.

Trust and data boundaries

User-provided input includes the spec string and the configuration files. Internal resolver code processes that input with the same privileges as the Spack process.

Package files are treated as data for metadata, but Spack package.py files are Python code. The resolver does not sandbox build scripts. A malicious package definition can execute arbitrary code during concretization if it manipulates Python-based package logic. Treat the package repository as trusted code.

Lockfiles are data, not code, but they encode assumptions about the configuration that produced them. Importing a lockfile generated with a development Spack build may be incompatible with older releases due to schema changes in the concrete spec format.

Operational checks

Determinism check. Generate two lockfiles from a clean environment and compare:

spack spec -l > spec1.lock
spack spec -l > spec2.lock
diff spec1.lock spec2.lock

Expected check: the files are identical when the configuration closure is unchanged.

Reproducibility check. Create a fresh Spack instance, load the lockfile and re-concretize:

spack env create -d testenv
spack lock --include-dependencies spec1.lock
spack concretize
spack spec --dependent

Expected check: the displayed DAG matches the original lockfile entries.

Missing dependency check:

spack find --missing

Run after concretization to surface packages without a buildable provider.

Binary cache integrity check. Before trusting a mirror:

spack buildcache keys --trust <mirror-url>
spack buildcache fetch <package>

Risk: relying on binary caches without verifying buildcache keys introduces supply-chain risk if a mirror is compromised.

Failure modes

Resolver conflict. Conflicting version constraints, e.g., requiring two incompatible versions of the same dependency, cause concretization to abort with a conflict message. This is the intended failure mode.

Missing compiler. If the concretized spec requires a compiler not listed in compilers.yaml, concretization aborts. Changing compiler configuration after a lockfile is created can cause spack install to fail or silently reuse incompatible binaries.

Network timeout. When a binary cache or mirror is unreachable, install or fetch operations time out. Concretization itself may succeed, but materialization fails.

Lockfile corruption or schema drift. Hash mismatches appear when the lockfile is edited manually or generated by a different Spack version. The practical check is to re-concretize and compare.

When the design would change

The current design assumes a single resolver process with trusted package code and configuration scopes that are fully local.

Design pressure points:

• External SAT solvers. Adopting an external solver would change the trust boundary and require a stable intermediate representation.
• Heterogeneous OS reproducibility. Reproducible builds across different operating systems would require capturing OS-level toolchain hashes and sandboxing.
• Strict sandboxing of package build scripts. Enforcing isolation during concretization and build would move execution out of the Spack process and require a more isolated execution environment.

Limitations to keep in mind: concretization results are sensitive to scope order, lockfiles are not portable across major Spack versions, and the resolver does not sandbox package code. Verify the configuration closure before sharing a lockfile.