Question
How can I prevent MPI initialization errors when running Ansys on a Slurm cluster with InfiniBand?
Zo Dey
0 reputation · 22 Aug 2021, 05:08 UTC
55.9K views
Loading…
0 reputation · 22 Aug 2021, 05:08 UTC
Deploying Ansys workloads on a Linux HPC cluster managed by Slurm and equipped with InfiniBand occasionally results in MPI initialization errors such as "multiple pkey found" or connection failures. The objective is to run Ansys simulations reliably without these MPI‑related obstacles, while respecting the cluster’s job scheduling constraints and ensuring the license server remains reachable.
What is the recommended method for determining the correct InfiniBand partition key (MPI_IB_PKEY) for a given Slurm partition? How should a Slurm job script be structured to set MPI_IB_PKEY before launching the Ansys solver, and how can this be validated without disrupting other MPI applications? Which logs or environment checks provide confidence that the InfiniBand fabric is being utilized correctly after the variable is applied?
35775 reputation · 22 Aug 2021, 15:51 UTC
When running Ansys simulations on a Slurm-managed HPC cluster with InfiniBand, MPI initialization errors such as "multiple pkey found" typically occur when the MPI library cannot determine the correct partition key (pkey) for the network fabric. To resolve this, you must identify the tenant-specific pkey and export it to the job environment.
ansyssrun --pty bash)./sys/class/infiniband/.Identify the InfiniBand device (usually mlx5_0) and read the available pkeys for the active port. The larger of the two values is the tenant key required for MPI communication.
cat /sys/class/infiniband/mlx5_0/ports/1/pkeys/0 cat /sys/class/infiniband/mlx5_0/ports/1/pkeys/1
Set the MPI_IB_PKEY or UCX_IB_PKEY environment variable before launching the solver. Using UCX_IB_PKEY is recommended for libraries utilizing the UCX framework (like HPC-X or OpenMPI).
#!/bin/bash #SBATCH --partition=your_partition #SBATCH --nodes=2 # Set the discovered pkey (example: 0x800b) export UCX_IB_PKEY=0x800b export MPI_IB_PKEY=0x800b # For Intel MPI, increase debug verbosity to verify fabric export I_MPI_DEBUG=5 # Launch Ansys solver srun ansys_solver_executable
Check the Slurm output log (slurm-<jobid>.out) for MPI debug strings. For Intel MPI, I_MPI_DEBUG=5 will provide details on the protocol and network used. Confirm that the logs do not mention "fallback to TCP" and explicitly reference the correct pkey.
If the job fails, remove the export lines from the script and resubmit. This restores the default MPI fabric discovery behavior to determine if the issue is specific to the manually assigned pkey.
Assumptions: This guide assumes the cluster uses Mellanox OFED drivers and that the InfiniBand device is named mlx5_0. It assumes the MPI library used by Ansys respects the UCX_IB_PKEY or MPI_IB_PKEY environment variables.
Use comments to ask for clarification. Post a solution as an answer.
5,960 reputation · 22 Aug 2021, 13:51 UTC
Ansys ships with a specific Open MPI build (commonly 4.0.2‑R1). If Slurm’s default MPI wrapper is active, the job may try to load the system MPI first, ignoring MPI_IB_PKEY or using a different IB stack. The result is a silent “multiple pkey” failure or fallback to TCP.
#!/bin/bash
#SBATCH --partition=highmem
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
# 1. Load the InfiniBand module so the driver is in the kernel
module load ib
# 2. Point the environment to the bundled MPI
export ANSYS_MPI_HOME=/opt/ansys/2026/mpi
export LD_LIBRARY_PATH=$ANSYS_MPI_HOME/lib:$LD_LIBRARY_PATH
# 3. Disable Slurm’s MPI wrapper – let Ansys’s MPI do the rank handling
srun --mpi=none ansys_solver -mpi
After the job starts, echo $LD_LIBRARY_PATH should list the bundled MPI directory first, and ibv_devinfo should show an active IB device. Check the job’s slurm-*.out for “MPI_Init” messages that reference the bundled MPI path.