Choosing Between PCI Passthrough Aliases and Placement-Based Scheduling for GPUs and FPGAs in OpenStack Nova

The Decision: How to Expose Specialized Hardware to Nova Instances

Operators adding GPUs, FPGAs, or SmartNICs to an OpenStack cloud face a concrete choice: continue using the legacy nova.conf [pci] alias mechanism with the PciPassthroughFilter, or adopt the Placement API with custom resource classes and standard traits. Since the Rocky release, Placement tracks both quantitative inventory (VCPU, MEMORY_MB, DISK_GB, and custom classes) and qualitative traits (e.g., HW_GPU_API_CUDA, HW_FPGA_INTEL_ARRIA10) per resource provider — compute nodes, nested NUMA nodes, or shared providers. The legacy alias approach matches only vendor_id/product_id at claim time and cannot express inventory limits, NUMA affinity, or trait requirements. New hardware types should use Placement; existing aliases remain functional but are deprecated for new designs.

Constraints That Shape the Choice

• OpenStack version: Placement-based scheduling requires Rocky or later; atomic claim of both PCI device and Placement allocation is guaranteed from Train onward.
• Hardware exclusivity: GPUs and FPGAs are typically non-shareable — allocation ratios must stay at 1.0.
• NUMA awareness: Workloads needing device-to-memory locality require NUMATopologyFilter and per-NUMA inventory, which only Placement supports.
• Operator tooling: Inventory and trait sync must be automated (Ansible, Puppet) because Placement does not auto-discover hardware changes.

Supported Options Compared

Capability	Legacy PCI Alias (nova.conf + PciPassthroughFilter)	Placement-Based (Custom Resource Classes + Traits)
Inventory tracking	None — matches device at claim time only	Per-resource-provider totals, reserved, min/max unit, step size, allocation ratio
NUMA affinity	Not expressible	Nested NUMA resource providers with separate inventory
Trait requirements	Not supported	Standard (HW_GPU_API_CUDA) or custom (CUSTOM_PHYSNET_DATACENTER) traits
Sharing prevention	Manual via alias device list	Allocation ratio = 1.0 + Placement allocation records
Scheduler weighing	Binary pass/fail	Aggregate isolation via trait:CUSTOM_HARDWARE_GPU, weighing plugins
Upgrade path	Continues to work; no new features	Recommended for all new hardware; mixed deployments supported since Train
Configuration surface	nova.conf [pci] alias + pci_passthrough_whitelist	placement.yaml (or osc placement CLI) + flavor extra_specs

Trade-Offs in Practice

The legacy alias path is simpler for a single GPU type on a homogeneous cluster: define an alias mapping vendor_id:product_id to a name, add the alias to a flavor extra_spec pci_passthrough:alias=gpu:1, and ensure PciPassthroughFilter is enabled. However, you cannot prevent two instances from landing on the same physical GPU if the alias lists multiple devices, nor can you express that a workload needs CUDA 12 support versus Vulkan only.

The Placement approach requires more upfront definition — custom resource class names must match ^CUSTOM_[A-Z0-9_]+$, traits must be created in the Placement database — but yields operational control: you set --total 4 --allocation-ratio 1.0 on the compute node's resource provider, attach HW_GPU_API_CUDA as a trait, and the scheduler will only place instances on providers reporting both the inventory and the trait. NUMA-aware workloads get correct placement because each NUMA node becomes a child resource provider with its own inventory slice.

Mixed deployments work: a compute node can report Placement inventory for new A100 GPUs while still satisfying legacy flavor requests for older K80s via PCI aliases. Nova claims both the PCI device (via libvirt) and the Placement allocation atomically, so there is no double-allocation window.

Concrete Implementation: Exposing NVIDIA A100 GPUs via Placement

Assume a compute node with four A100 GPUs, each bound to vfio-pci and mediated via mdev (vGPU) or direct passthrough. Kernel ≥ 4.10, QEMU ≥ 3.1, and NVIDIA vGPU driver are present. The following steps are run on the controller (for Placement API calls) and the compute node (for libvirt mediation).

1. Define the Custom Resource Class and Standard Trait

# On controller (admin credentials)
openstack --os-placement-api-version 1.38 resource class create CUSTOM_GPU_NVIDIA_A100
openstack --os-placement-api-version 1.38 trait create HW_GPU_API_CUDA
# HW_GPU_API_CUDA exists in os-traits; create only if missing

Required permission: admin role in the service project. The resource class name must start with CUSTOM_ and contain only uppercase letters, digits, and underscores.

2. Register Inventory and Traits on the Compute Node's Resource Provider

# On controller, using the compute node's RP UUID
RP_UUID=$(openstack --os-placement-api-version 1.38 resource provider list --name compute-01 -f value -c uuid)
openstack --os-placement-api-version 1.38 resource provider inventory set \
  --resource-class CUSTOM_GPU_NVIDIA_A100 \
  --total 4 --reserved 0 --min-unit 1 --max-unit 4 --step-size 1 --allocation-ratio 1.0 \
  $RP_UUID
openstack --os-placement-api-version 1.38 resource provider trait add $RP_UUID HW_GPU_API_CUDA

Run after each kernel driver change or firmware update. Automate with configuration management to keep placement.yaml (or the equivalent provider config) in sync. Risk: stale inventory causes allocation failures or overcommit if --total exceeds physical devices.

3. Create a Flavor That Requests the Resource and Trait

openstack flavor create --vcpus 8 --ram 32768 --disk 100 gpu-a100
openstack flavor set --property "resources:CUSTOM_GPU_NVIDIA_A100=1" \
  --property "trait:HW_GPU_API_CUDA=required" gpu-a100

Keys resources: and trait: are case-sensitive and must be lowercase. The flavor now requires one A100 GPU and a provider advertising CUDA support.

4. Configure Libvirt Mediation on the Compute Node

Edit /etc/nova/nova.conf (or the appropriate libvirt section) to enable mediated devices:

[libvirt]
# For mdev vGPU
gpu_mdev_types = nvidia-222  # example mdev type from 'virsh nodedev-list --cap mdev_type'
# For direct passthrough, ensure vfio-pci binds the GPU PCI IDs
# and add to pci_passthrough_whitelist if legacy alias also used

Restart nova-compute after changes. Verify mdev types: virsh nodedev-list --cap mdev_type.

5. Launch a Test Instance and Validate Allocation

# On controller
openstack server create --flavor gpu-a100 --image cirros --network private test-gpu
# Wait for ACTIVE
SERVER_UUID=$(openstack server show test-gpu -f value -c id)
openstack --os-placement-api-version 1.38 resource provider allocation show $SERVER_UUID

Expected output shows an allocation record for CUSTOM_GPU_NVIDIA_A100=1 against the compute node's RP (or its NUMA child). On the compute node, inspect libvirt XML:

virsh dumpxml $SERVER_UUID | grep -A5 ' for vfio-pci
# or  for mediated device

6. Workload Validation Inside the Instance

# SSH into instance, then
nvidia-smi
# Should list the assigned GPU with driver version and compute capability

If the trait HW_GPU_API_CUDA was requested, the instance must see a CUDA-capable device. For FPGA or SmartNIC, substitute vendor-specific diagnostics (e.g., fpgainfo, ethtool -i).

Verification Checklist

1. Placement API version: openstack --os-placement-api-version 1.38 resource provider list --allocations returns 200 OK.
2. Inventory reported: openstack resource provider inventory list $RP_UUID | grep CUSTOM_GPU shows total=4, allocation_ratio=1.0.
3. Instance launches: openstack server create ... reaches ACTIVE; allocation consumer appears in Placement.
4. Libvirt passthrough confirmed: virsh dumpxml shows hostdev with correct PCI/mdev address.
5. Workload runs: nvidia-smi (or vendor tool) reports the device inside the guest.
6. Upgrade checks clean: placement-status upgrade check and nova-status upgrade check report no errors.

Limitations and Operational Notes

• Allocation ratio > 1.0 on custom resource classes for exclusive devices is not recommended — it defeats the purpose of Placement tracking and leads to overcommit errors at claim time.
• Inventory drift is the primary failure mode: if a GPU fails or is replaced, placement.yaml or the automated sync must update --total before the next scheduler run.
• Scheduler filters: ensure FilterScheduler.enabled_filters includes PlacementFilter (default) and NUMATopologyFilter for NUMA-aware workloads. PciPassthroughFilter is deprecated for new designs but may remain for legacy flavors.
• Trait namespace: prefer standard traits from os-traits (e.g., HW_GPU_API_VULKAN, HW_FPGA_INTEL_ARRIA10). Custom traits must also use the CUSTOM_ prefix.
• Rollback: removing a custom resource class or trait from a provider is a state-changing operation. To roll back, delete the inventory (resource provider inventory delete) and remove the trait (resource provider trait delete), then restart nova-compute.

When to Stay with PCI Aliases

Legacy aliases remain acceptable for:

• Existing flavors that cannot be recreated (e.g., third-party templates).
• Hardware that cannot be modeled as discrete inventory (e.g., a pool of identical NICs where any device satisfies the request).
• Environments pinned to pre-Rocky OpenStack versions.

For all new GPU, FPGA, or SmartNIC deployments, the Placement path is the supported, future-proof choice.