Diagnosing Kubeflow Pipelines Runs Stuck in Pending on Kubernetes

When a Kubeflow Pipelines Run Never Leaves Pending

A Kubeflow Pipelines run that stays Pending is almost always a Kubernetes scheduling or storage problem, not a defect in the pipeline code. The fastest way to unblock it is to read the pod events in the run's namespace and match the exact event message to the resource that Kubernetes could not allocate. This guide covers the recognizable symptoms, the ordered checks that isolate the cause, and the targeted fixes—plus the point at which you should hand the problem to cluster or platform owners.

Recognize the Condition

The run detail page in the Kubeflow Pipelines UI shows STATUS Pending with no step logs. Listing the run's pods shows them in STATUS Pending with no container statuses. The decisive signal is in the Kubernetes events: they show FailedScheduling or FailedBinding, not ImagePullBackOff, CrashLoopBackOff, or a completed step that failed. If you see image pull errors or container crashes, the pods scheduled successfully and this guide does not apply.

Map the Cause to the Diagnostic Signal

Use the exact event text from kubectl describe pod to select the matching row.

Likely cause	Diagnostic signal
Not enough CPU/memory, or wrong nodeSelector/tolerations	FailedScheduling: 0/N nodes are available: insufficient cpu
PVC unbound, or wrong storageClassName/accessModes	FailedBinding: no persistent volumes available for this claim
Missing pipeline-runner service account or RBAC	Pod events show serviceaccount not found or forbidden
Admission webhook or sidecar injection failure	Pod Pending with webhook denied or failed to create pod sandbox
Namespace ResourceQuota exhausted	FailedCreate: exceeded quota

Ordered Checks

1. Confirm scope. Determine whether only one run is affected or all runs in the namespace. Run this in your terminal with cluster access:

   kubectl get pods -n NAMESPACE | grep Pending

   Expected check: if every pod in the namespace is Pending, suspect a namespace-wide quota, RBAC, or webhook issue. If only the pods for one run ID are Pending, suspect that run's resource requests or storage configuration.

2. Inspect the run pods. Identify the pod names for the stuck run, then describe one to read its events:

   kubectl describe pod -n NAMESPACE POD_NAME

   Expected check: scroll to the Events section at the bottom. You are looking for FailedScheduling, FailedBinding, FailedCreate, or service-account errors. This command is read-only and requires only get/describe permissions on pods in the namespace.

3. Filter events by pod. For a tighter view, use a field selector:

   kubectl get events -n NAMESPACE --field-selector involvedObject.name=POD_NAME

   Expected check: the most recent event should match one of the signals in the table above.

4. Check quota and limits. Compare used versus hard capacity:

   kubectl get resourcequota -n NAMESPACE -o yaml

   Expected check: if used equals or exceeds hard for cpu, memory, or pods, the quota is blocking new pod creation.

5. Check PVC state and storage class. Verify that the pipeline's volume claims can bind:

   kubectl get pvc -n NAMESPACE
   kubectl get sc

   Expected check: PVCs should be Bound. If a PVC is Pending, confirm that the storageClassName it references exists in the kubectl get sc output and that the class supports the requested accessModes.

6. Verify images and pipeline root. Confirm that the component images referenced by the pipeline exist and are pullable from the cluster nodes, and that the pipeline root location is reachable. This rules out artifact or registry problems that can appear after scheduling succeeds.

Apply the Fix That Matches the Event

• FailedScheduling for insufficient resources. Lower the component's resources.requests in the pipeline definition, or add tolerations and a nodeSelector if the intended node pool is tainted. If the namespace quota is the true ceiling, request an increase from the namespace owner.

• FailedBinding for storage. Correct the storageClassName and accessModes in the pipeline or volume claim so they match an available StorageClass. Ensure the class has a working dynamic provisioner, or supply a pre-bound PersistentVolume.

• Service account or RBAC errors. Create the pipeline-runner ServiceAccount and the RoleBinding that Kubeflow Pipelines expects in the namespace. Verify the exact names against your installed manifests, because defaults differ across versions and distributions.

• Admission webhook denial. Coordinate with the team that owns the webhook to fix the rejecting policy. If your environment permits it, bypass the webhook for the pipeline-runner pods, but treat this as a temporary workaround.

• Exceeded quota. Delete pods from runs that are already Completed or Failed to reclaim quota, then compare used versus hard again:

  kubectl delete pod -n NAMESPACE POD_NAME

  Risk: only delete pods from finished runs. Deleting a Pending pod from an active run may cause the controller to recreate it with the same unsatisfiable request. Do not edit a cluster-wide ResourceQuota to fix a single run; that affects other tenants and may be reverted by GitOps.

When to Escalate to Cluster or Platform Owners

Stop troubleshooting in the application namespace and escalate when the root cause is outside your control:

• Nodes are NotReady or schedulable capacity is effectively zero.
• The default StorageClass or its CSI driver is missing or broken.
• An admission webhook owned by another team rejects all pods in the namespace.
• The same run fails identically on a cluster you believe is healthy.

When you escalate, include the namespace, run ID, pod name, and the exact event message from kubectl describe pod.

Verify the Fix and Confirm Stability

Reproduce with a minimal pipeline that requests the same resources and storage class. If it schedules and completes, the original run's requests or namespace state were the cause. After applying a fix, confirm the run leaves Pending:

kubectl get pods -n NAMESPACE -l pipelines.kubeflow.org/run-id=RUN_ID

Expected check: pods transition to Running or Completed, and the run detail exposes step logs. Rerun the same pipeline to confirm the fix is stable. Finally, validate quota headroom by comparing kubectl describe resourcequota output before and after cleanup, and confirm that no new FailedScheduling events appear for later runs.

Limitations and Version Sensitivity

This guide assumes Kubeflow Pipelines on a Kubernetes cluster where you can inspect pods, events, PVCs, and quotas. Multi-user isolation, the underlying workflow engine (Argo Workflows or Tekton), and sidecar or webhook behavior vary by Kubeflow version and distribution. Managed offerings may restrict node or quota changes. If pods schedule but steps never start, check the workflow controller and artifact repository before blaming scheduling. Avoid applying manifests from memory; component names, CRDs, and default service accounts differ across releases. Verify every change against your installed manifests before modifying cluster policy.