Fix Kubeflow GPU Scheduling Failures

When Kubeflow Pipelines can’t schedule GPU‑enabled pods, the scheduler emits <code>Unschedulable</code> events. This guide walks through common causes, ordered diagnostics, fixes, and escalation steps—covering device plugin installation, quotas, pod specs, node taints, and kubelet feature gates.

Recognizing the Problem

When a Kubeflow Pipeline run that requests GPUs never reaches the Running state, the most common symptom is an Unschedulable status with a FailedScheduling event. Typical messages include Insufficient nvidia.com/gpu or Node has insufficient GPU resources. These errors indicate that the Kubernetes scheduler cannot find a node that satisfies the pod’s GPU request.

Common Causes

The failure can stem from several layers: node labeling, resource plugin installation, quota limits, pod specification errors, or cluster configuration. Understanding the root cause requires a systematic approach.

Diagnostic Table

Cause	Diagnostic Check	Typical Fix
GPU nodes missing the NVIDIA device plugin or labels	kubectl get nodes -o wide \| grep nvidia	Install or update the NVIDIA device plugin; label nodes with `nvidia.com/gpu.present=true`
GPU quota exhausted or misconfigured	kubectl describe quota -n namespace	Adjust or remove quota limits; request quota increase from cluster admin
Pod requests GPUs incorrectly (wrong key or unit)	kubectl describe pod name \| grep -i gpu	Correct `resourceName` to `nvidia.com/gpu` and use integer values
Node drivers or NVIDIA Container Toolkit missing	ssh into node && nvidia-smi	Install CUDA drivers and the NVIDIA Container Toolkit
Cluster kubelet missing GPU feature gates	kubectl get nodes -o yaml \| grep featureGates	Add `--feature-gates=DevicePlugins=true` to kubelet
Pipeline components not passing GPU requests	Inspect pipeline YAML for `resources` section	Configure the component to request GPUs explicitly
GPU nodes tainted	kubectl describe node node \| grep Taints	Add matching tolerations to the pod spec

Ordered Checks

Verify Node GPU Availability
Run:
```
kubectl get nodes -o wide | grep nvidia
```
Each GPU node should list a label such as nvidia.com/gpu.present=true and a Capacity entry for nvidia.com/gpu (e.g., nvidia.com/gpu: 4). If the label or capacity is missing, install the device plugin for your Kubernetes version.
Check the NVIDIA Device Plugin Deployment
For Kubernetes <1.24:
```
kubectl get deployment nvidia-device-plugin -n kube-system -o yaml
```
For 1.24+:
```
kubectl get daemonset nvidia-device-plugin -n kube-system -o yaml
```
Confirm the plugin is running and the container image matches the cluster’s GPU driver version.
Inspect GPU Quotas
In the namespace where the pipeline runs:
```
kubectl describe quota -n namespace
```
Look for limits.nvidia.com/gpu. If the used value equals the hard limit, the scheduler cannot place additional pods.
Validate Pod GPU Requests
Describe the pod that failed to schedule:
```
kubectl describe pod pod-name | grep -i gpu
```
Ensure the spec uses resourceName: nvidia.com/gpu and the value is an integer (e.g., requests: {nvidia.com/gpu: 1}). Deprecated keys like nvidia.com/gpu: 1.0 or nvidia.com/gpu: "1" will cause the scheduler to reject the pod.
Check Node Taints and Pod Tolerations
List taints on GPU nodes:
```
kubectl get nodes -o json | jq '.items[].spec.taints'
```
If a node has a taint such as nvidia.com/gpu: NoSchedule, the pod must include a matching toleration:
```
tolerations:
- key: nvidia.com/gpu
  operator: Exists
  effect: NoSchedule
```
Verify Kubelet Feature Gates
Check kubelet flags on the node:
```
ps -ef | grep kubelet | grep -i feature-gates
```
The flag --feature-gates=DevicePlugins=true must be present for GPU scheduling to work in Kubernetes <1.24. For newer versions, ensure --feature-gates=DevicePlugins=true is still set if required.
Confirm Driver and Toolkit Compatibility
SSH into a GPU node and run:
```
nvidia-smi
```
Failure indicates a driver or toolkit mismatch. Reinstall the drivers and the NVIDIA Container Toolkit that matches the CUDA version used by your containers.

Fixes Tied to Findings

Install or Upgrade the Device Plugin

For Kubernetes <1.24:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.12.0/nvidia-device-plugin.yml

For 1.24+, use the CRI‑Containerd integration:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/master/nvidia-device-plugin.yaml

After deployment, verify the daemonset pods are running and the nodes report GPU capacity.

Adjust or Remove GPU Quotas

To increase the limit:

kubectl patch quota gpu-quota -n namespace -p '{"spec":{"hard":{"nvidia.com/gpu":"8"}}}'

Or delete the quota if it’s unnecessary:

kubectl delete quota gpu-quota -n namespace

Correct Pod Spec
Update the pipeline component YAML:
```
resources:
  limits:
    nvidia.com/gpu: 1
  requests:
    nvidia.com/gpu: 1
```
Re‑submit the pipeline run after the change.

Add Tolerations

Include the toleration in the pod template:

spec:
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

Enable Kubelet Feature Gates
Modify the kubelet startup script or systemd unit to add:
```
--feature-gates=DevicePlugins=true
```
Restart kubelet and verify the node again shows GPU capacity.

Upgrade Drivers and Toolkit

On each GPU node, install the driver matching the CUDA version required by your containers, then install the NVIDIA Container Toolkit:

curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu20.04/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker

Escalation Criteria

If all checks above return no actionable findings, consider the following escalation path:

Cluster‑Wide GPU Configuration – Contact the cluster administrator to confirm that GPU nodes are part of the node pool and that no global taints or node selectors block scheduling.
Infrastructure Limits – Verify that the underlying cloud provider or on‑prem infrastructure allows adding GPU nodes or scaling the node pool.
Compatibility with Kubeflow Version – Ensure the Kubeflow release you use supports the Kubernetes version and GPU plugin version. Refer to the Kubeflow compatibility matrix.

Practical Verification

After applying fixes, run a lightweight GPU test job:

cat <<'EOF' | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-test
spec:
  template:
    spec:
      containers:
      - name: cuda
        image: nvidia/cuda:11.8.0-base-ubuntu20.04
        command: ["nvidia-smi"]
        resources:
          limits:
            nvidia.com/gpu: 1
      restartPolicy: Never
  backoffLimit: 0
EOF

kubectl logs job/gpu-test

Successful output of nvidia-smi confirms that GPU scheduling is functional. If the job still fails, inspect the pod events for new FailedScheduling reasons.

Limitations and Caveats

GPU quota changes affect all workloads in the namespace; coordinate with other teams to avoid resource starvation.
Modifying node labels or taints requires cluster‑admin privileges and may impact other deployments.
Driver and toolkit versions must match the CUDA runtime used by your containers; mismatches can cause runtime errors that are not caught by the scheduler.
Device plugin deprecation in newer Kubernetes releases means older manifests may fail silently; always check the plugin documentation for your Kubernetes version.

By following the ordered checks and applying the corresponding fixes, most GPU scheduling failures in Kubeflow Pipelines can be resolved without escalated support. Consistent verification after each change ensures that the underlying issue is truly addressed.

Diagnosing GPU Pod Scheduling Failures in Kubeflow Pipelines