How can I prevent MPI initialization errors when running Ansys on a Slurm cluster with InfiniBand?

When running Ansys simulations on a Slurm-managed HPC cluster with InfiniBand, MPI initialization errors such as "multiple pkey found" typically occur when the MPI library cannot determine the correct partition key (pkey) for the network fabric. To resolve this, you must identify the tenant-specific pkey and export it to the job environment.

Prerequisites

• Access to a compute node via Slurm (e.g., using srun --pty bash).
• Root or user permissions to read /sys/class/infiniband/.
• An MPI implementation compatible with InfiniBand (e.g., HPC-X, Intel MPI, or OpenMPI).

Step-by-Step Implementation

1. Discover the Partition Key:

   Identify the InfiniBand device (usually mlx5_0) and read the available pkeys for the active port. The larger of the two values is the tenant key required for MPI communication.

   cat /sys/class/infiniband/mlx5_0/ports/1/pkeys/0
   cat /sys/class/infiniband/mlx5_0/ports/1/pkeys/1

2. Structure the Slurm Job Script:

   Set the MPI_IB_PKEY or UCX_IB_PKEY environment variable before launching the solver. Using UCX_IB_PKEY is recommended for libraries utilizing the UCX framework (like HPC-X or OpenMPI).

   #!/bin/bash
   #SBATCH --partition=your_partition
   #SBATCH --nodes=2
   
   # Set the discovered pkey (example: 0x800b)
   export UCX_IB_PKEY=0x800b
   export MPI_IB_PKEY=0x800b
   
   # For Intel MPI, increase debug verbosity to verify fabric
   export I_MPI_DEBUG=5
   
   # Launch Ansys solver
   srun ansys_solver_executable
   

3. Verification:

   Check the Slurm output log (slurm-<jobid>.out) for MPI debug strings. For Intel MPI, I_MPI_DEBUG=5 will provide details on the protocol and network used. Confirm that the logs do not mention "fallback to TCP" and explicitly reference the correct pkey.

4. Rollback:

   If the job fails, remove the export lines from the script and resubmit. This restores the default MPI fabric discovery behavior to determine if the issue is specific to the manually assigned pkey.

Assumptions: This guide assumes the cluster uses Mellanox OFED drivers and that the InfiniBand device is named mlx5_0. It assumes the MPI library used by Ansys respects the UCX_IB_PKEY or MPI_IB_PKEY environment variables.