Diagnosing Kubeflow Notebook ImagePullBackOff Failures

The Problem: Notebook Stuck in ImagePullBackOff

When you launch a Kubeflow Notebook and the pod never reaches Running, Kubernetes often reports ImagePullBackOff. This means the kubelet on the node tried to fetch the container image and failed, then backed off before retrying. The notebook server cannot start until the image is present.

Recognizable Condition

• Pod status is ImagePullBackOff or ErrImagePull.
• kubectl get notebooks shows the Notebook custom resource, but its associated pod restarts repeatedly.
• The notebook server URL never becomes available in the Kubeflow UI.

A Notebook custom resource is the Kubeflow object that defines a notebook server. ImagePullBackOff is a Kubernetes pod status meaning the container image could not be pulled and the kubelet will retry with increasing delay.

Common Causes and Diagnostic Signals

Observed condition	Likely cause	Where to confirm
Image name or tag does not exist in registry	Typo in image reference or tag	kubectl describe pod events
Registry credentials missing or rejected	No imagePullSecret, or secret mismatched to registry	kubectl get secret and pod spec
Repeated pulls for a cached image	imagePullPolicy set to Always	Pod spec and kubelet behavior
Node cannot reach registry host or port	Network policy, DNS failure, or proxy block	Node-level or cluster-level connectivity test

Ordered Diagnostic Checks

Check 1: Confirm the pod and recent events

Run from any machine with kubectl configured for the cluster. You need read access to the namespace.

kubectl get pods -n <namespace> | grep <notebook-name>
kubectl describe pod <pod-name> -n <namespace>

Expected check: the Events section shows Failed to pull image or a registry error followed by the exact image reference. This is read-only and carries no risk.

Check 2: Inspect the Notebook custom resource image

kubectl get notebook <notebook-name> -n <namespace> -o yaml

Expected check: under spec.template.spec.containers[0].image, compare the image and tag exactly against the registry repository. A digest such as @sha256:... is valid only if that digest exists in the registry.

Check 3: Verify the image pull secret

kubectl get secret <secret-name> -n <namespace> -o yaml

Expected check: the secret type is kubernetes.io/dockerconfigjson and its data contains authentication for the registry host in the image reference. Also confirm the Notebook custom resource lists the secret under spec.template.spec.imagePullSecrets. Do not share secret contents in public channels.

Check 4: Check image pull policy

Review the same Notebook custom resource output. If the image is already cached on the node, IfNotPresent avoids repeated pulls. Always forces a fresh pull and can expose transient registry outages. Changing this requires recreating the pod.

Check 5: Test registry reachability from the cluster

Launch a short-lived test pod in the same namespace. You need permission to create pods.

kubectl run registry-test --image=curlimages/curl --rm -i --restart=Never -- curl -sS -o /dev/null -w "%{http_code}" https://<registry-host>/v2/

Expected check: 200 means the registry is reachable. 401 means network works but authentication is wrong. Connection refused or timeout indicates a network block. Delete the test pod after testing.

Fixes Tied to Findings

• Bad image reference: Correct the image or tag in the Notebook custom resource and reapply. Kubeflow creates a new pod.
• Missing or wrong secret: Create or update the imagePullSecret in the namespace, add it to the Notebook custom resource's imagePullSecrets, and reapply. Because secrets are not injected into running pods, delete the old pod so the controller recreates it.
• Pull policy mismatch: Change imagePullPolicy to IfNotPresent for stable cached images, or keep Always only when you actively track latest tags. Reapply and recreate the pod.
• Network block: Ask cluster administration to open egress to the registry host and port, commonly 443, adjust network policies, or configure the cluster proxy. Then recreate the notebook pod.

Rollback

If a corrected image still fails, revert the Notebook custom resource image field to the previous known-good value and reapply. The controller starts new pods using the earlier image. Diagnostic commands themselves are read-only and do not require rollback.

Escalation Criteria

Escalate to platform or registry administration when:

• The image exists and is public, but every node in the cluster fails to reach the registry.
• The imagePullSecret is correct, yet the registry returns authentication errors for all accounts, suggesting a registry-side outage.
• You cannot modify network policies or secrets due to restricted namespace permissions.

Verification and Limitations

After applying a fix, watch the pod:

kubectl get pod <pod-name> -n <namespace> -w

Expected result: the status moves from ContainerCreating to Running. If it returns to ImagePullBackOff, repeat Check 1 and read the new event message.

Limitations: this guide covers image retrieval failures only. If the image pulls successfully but the notebook container crashes immediately, the cause is likely the entrypoint, dependencies, or Kubeflow profile configuration, not ImagePullBackOff. This guide does not cover GPU driver errors or node resource allocation failures.