Speeding Up Repeated HPC Builds with Spack’s Build Cache

The problem: rebuilding the same stack on every node

In large HPC clusters, each compute node often starts from a clean environment and rebuilds the entire software stack from source. This repeats the same compilation work many times, wasting CPU cycles, energy, and user time. While reproducibility is essential, repeating identical builds is unnecessary when the underlying spec, compiler, and flags have not changed.

Thesis: enable Spack’s build cache to reuse binaries safely

Spack can store a tarball of each installed package, keyed by the full spec hash, in a configurable location. When a later install requests the same spec, Spack unpacks the cached tarball instead of recompiling, delivering the same binary with guaranteed bit‑for‑bit reproducibility.

How the build cache works

Spack’s build cache is activated per scope (e.g., scopes.yaml). When enabled, Spack:

1. Computes a hash that includes the package name, version, compiler, variants, dependencies, and any relevant flags.
2. Looks for a .spack tarball matching that hash in the cache directory.
3. If found, it extracts the tarball to the install path; otherwise it builds from source and stores the resulting tarball for future use.

The cache is agnostic to the client node; any node with access to the shared location can benefit.

Worked example: sharing HDF5 and ParaView builds

Assume a cluster with a shared filesystem mounted at /mnt/spack-cache. We will configure a shared cache, install hdf5 once, and then reuse it when installing paraview.

1. Configure the cache

Add or edit etc/spack/defaults/scopes.yaml:

cache:
  build_cache:
    enabled: true
    path: /mnt/spack-cache
    # optional: limit total size (in GiB)
    # max_size: 500

Run spack location -r to confirm Spack reads the file; no output is required, just ensure no errors appear.

2. Install the first package

From any node with access to the cache:

spack install hdf5%gcc@11.2.0 ^zlib

Spack will build HDF5 from source and, upon success, place a tarball named something like hdf5-1.12.2-gcc-11.2.0-.spack under /mnt/spack-cache.

3. Verify the cache was populated

Check that the install path resides inside the cache:

spack location -p hdf5%gcc@11.2.0

The returned path should begin with /mnt/spack-cache/ followed by the spec hash and the install directory.

4. Install a dependent package

Now install ParaView, which depends on HDF5:

spack install paraview%gcc@11.2.0 ^hdf5

Spack will locate the cached HDF5 tarball, extract it, and skip recompiling HDF5. Only ParaView’s own build steps run.

5. Confirm cache reuse

After the install, examine cache statistics:

spack stats

Look for a non‑zero “build cache hits” count. Additionally, spack cache list will show the stored tarballs, including the HDF5 entry.

Trade‑offs and limitations

While the build cache can dramatically cut compile times, consider the following:

• Cache coherency: Changing a compiler version, adjusting cflags, or toggling a variant alters the spec hash, producing a miss. Teams must coordinate when to update shared compiler modules to avoid fragmentation.
• Storage overhead: Each unique spec generates a tarball. In environments with many variant combinations, the cache can grow quickly. Implement a cleanup policy (e.g., spack gc or size‑based pruning) to reclaim space.
• Network latency: If the cache resides on a slow shared filesystem (e.g., NFS with high latency), the time to fetch and unpack a tarball may offset compilation savings. Benchmark with /usr/bin/time -p spack install <pkg> on a representative workload to decide.

Actionable closing steps

To start using the build cache in your HPC center:

1. Choose a shared, reliable filesystem path accessible from all compute nodes.
2. Enable the cache in scopes.yaml (or a site‑wide packages.yaml scope) with enabled: true and the chosen path.
3. Run a baseline install of a core dependency (e.g., hdf5 or openmpi) to populate the cache.
4. Monitor hits/misses via spack stats after a week of typical workloads; adjust compiler module versions or variant policies to improve hit rates.
5. Periodically run spack gc --delete‑bytes‑older‑than 30d or set a size limit to control disk usage.

By following these steps, teams can retain Spack’s reproducibility guarantees while saving valuable compute cycles for actual science rather than repeated recompilation.