Installing OpenFOAM with Spack Using Multiple Compilers and MPI Variants

Desired Outcome

Build and install OpenFOAM twice with Spack, once using gcc and OpenMPI, and once using clang and MPICH. Each build must be isolated in its own prefix, and the runtime environment must load the correct compiler and MPI libraries.

Prerequisites

• Spack 0.20 or newer installed in the user’s home directory (~/.spack).
• At least two compilers available on the system and discoverable by Spack. Typical commands to install them on a Linux system:

  # Example for Ubuntu
  sudo apt-get install gcc-12 g++-12 clang-15
  

  Spack will auto‑detect these after a spack compiler find.
• At least two MPI implementations installed and discoverable by Spack. For example:

  # Example for Ubuntu
  sudo apt-get install openmpi-bin libopenmpi-dev mpich libmpich-dev
  

• Administrative access to install packages if the system requires it, otherwise use user‑space install.
• Sufficient disk space in ~/.spack/opt/spack for two separate OpenFOAM prefixes.

Procedure

1. Verify compiler catalog

   spack compiler find
   

   Check that gcc@12 and clang@15 appear. If missing, add them manually with spack compiler add.
2. Verify MPI catalog

   spack spec mpich
   

   Ensure that mpich@3.4.2 and openmpi@4.1.6 are resolvable.
3. Install OpenFOAM with gcc and OpenMPI

   spack install openfoam@develop%gcc ^openmpi@4.1.6
   

   This creates a prefix under ~/.spack/opt/spack/.../openfoam@develop%gcc^openmpi@4.1.6.
4. Install OpenFOAM with clang and MPICH

   spack install openfoam@develop%clang ^mpich@3.4.2
   

5. Confirm both installs exist

   spack find openfoam
   

   You should see two entries with distinct prefixes.
6. Load the first spec and check environment

   spack load openfoam@develop%gcc ^openmpi@4.1.6
   echo $SPACK_CC
   

   The output should be gcc.
7. Run a simple OpenFOAM test case

   cd $HOME/OpenFOAM-$(spack find -l openfoam | grep -m1 '^openfoam' | awk '{print $2}')/tutorials/liggghts/...
   # Example command to launch with mpirun
   mpirun -np 1 simpleFoam
   

   Verify that the MPI implementation matches the spec by inspecting ldd $(which simpleFoam).

Expected Checks

• Installation prefixes: ~/.spack/opt/spack/.../openfoam@develop%gcc^openmpi@4.1.6 and .../openfoam@develop%clang^mpich@3.4.2 should exist.
• Environment modules: After spack load, module list should show the correct compiler and MPI paths.
• Runtime linking: ldd $(which simpleFoam) should list libmpi.so from the chosen MPI implementation.
• Test case execution: The OpenFOAM case should finish without errors and the output logs should reference the expected MPI implementation.

Recovery Options

• Build failure: Inspect the build log at ~/.spack/build-env//build.log to locate compile or link errors.
• Conflicting specs: If Spack picks the wrong compiler, explicitly name the compiler in the spec string: gcc@12 or clang@15.
• Clean and retry: Remove the problematic build with spack clean -a and re‑run the spack install command.
• Isolate build directories: Adjust build_stage:paths in ~/.spack/config.yaml to avoid clashes when building many specs concurrently.
• Reduce parallelism: Use spack install --jobs 1 if the system runs out of memory or CPU during the build.

By following these steps, you achieve reproducible, isolated builds of OpenFOAM with different compiler and MPI combinations, and you have clear diagnostics and recovery paths for common build issues.