Spack environment recovery after a failed upgrade: lockfile re-concretization vs. preserving the old environment view

Direct Answer

Recovery should be based on keeping the pre-upgrade environment untouched until the upgrade is verified, not on restoring spack.lock and reinstalling. The lockfile alone does not guarantee bit-for-bit reproducibility across Spack releases because package.py concretizer logic and hash algorithms drift. Pinning the Spack release improves reproducibility but does not eliminate hash differences. The safe ordering is: verify the old specs are still installed, do not run spack gc or global spack clean until the new environment builds and passes tests, then clean up the fallback.

Why the Lockfile Is Not a Rollback Guarantee

spack.lock records the concretized DAG at the moment of installation. When you later run spack install against that lockfile on a different Spack version (or even a different minor release of the same major version), the concretizer may resolve dependencies differently because:

• Package version or variant directives changed in package.py
• Concretizer preference ordering or conflict resolution logic was updated
• Hash algorithms for spec identification evolved (e.g., spack hash output format)

Result: the reinstalled packages can have different hashes, breaking binary compatibility with any downstream packages that were built against the original hashes.

Pinning the Spack Release Helps, But Is Not Sufficient

Checking in the exact Spack commit (e.g., via spack.yaml spack: section or a spack.lock metadata field) ensures the same concretizer code runs. However:

• If the original lockfile was created with Spack 0.20.0 and you restore on 0.20.1, a bug-fix in a single package.py can change a dependency version.
• External package repositories or overlays may have been updated independently.
• The lockfile does not capture the package.py file hashes themselves.

For true reproducibility you need the exact Spack source tree (or a container image) that produced the original lockfile.

Safe Ordering: Preserve the Fallback First

1. Before starting the upgrade, activate the current environment and run:

   spack env activate myenv
   spack find --format="{name}/{version}/{hash:7}"  # record installed specs
   spack spec --long  # verify concretization matches lockfile

2. Do not run spack gc or spack clean (without --spec) at any point before the new environment is verified. gc removes any package not referenced by an active spec in any environment; a failed upgrade leaves the old specs unreferenced and they will be deleted.
3. Perform the upgrade in a separate environment directory (copy spack.yaml, edit specs, spack concretize, spack install). The original environment's install tree remains untouched.
4. After the new environment builds and passes your validation tests, you may:
   • spack env deactivate the old environment
   • spack gc -y to reclaim space (now the old specs are truly unreferenced)
   • spack clean --spec to remove only the old build stages if desired

Verification Step Before Cleanup

Run this one-liner against the original environment to confirm the fallback is intact:

spack env activate myenv && spack find -l --format="{name}/{version}/{hash:7}" | grep -v "^==>"

If the expected core specs (compiler, MPI, etc.) appear with their original hashes, the fallback is safe. Only then proceed with gc/clean.

One Diagnostic Detail That Changes the Recommendation

Are you using multiple environments that share the same install tree (i.e., the same spack.yaml install_tree:root)? If yes, spack gc will only remove packages unreferenced by all environments, so the old specs may survive longer. If no (each environment has its own install tree), the fallback is entirely dependent on not running gc in that environment.