Diagnosing Image Pull Failures in k3OS Nodes

Identify the Failure Mode

When a pod stays in ImagePullBackOff, the underlying cause is usually one of the following.

Error observed	Likely cause	Where to look
401 Unauthorized / 403 Forbidden	Missing or wrong registry credentials	/etc/containers/registries.conf
404 Not Found	Typo in image name or tag	Deployment manifest
Connection timeout / No route	Network blockage (firewall, DNS, systemd‑networkd)	Node network stack
Manifest not found / digest mismatch	Architecture mismatch or unsupported tag	Registry metadata

Step 1: Inspect containerd logs

Because k3OS uses containerd as the CRI runtime, the kubelet’s generic message hides the real HTTP response. Check the journal for the containerd unit:

journalctl -u containerd -n 100 --no-pager

Search for lines with failed to pull or unauthorized. A line containing context deadline exceeded points to a network timeout rather than auth.

Step 2: Verify network reachability

If the logs suggest a timeout, test the registry endpoint directly from the node:

# Replace with your registry host
curl -v https://registry.example.com/v2/

A successful TLS handshake (HTTP 200 or 401) confirms that outbound traffic is allowed. If the command hangs or returns Could not resolve host, inspect the network configuration:

systemctl status systemd-networkd

Ensure the interface is UP and has an IPv4/IPv6 address.

Step 3: Validate registry authentication

When network is reachable but you see 401/403, the credentials in /etc/containers/registries.conf are likely incorrect or missing. Remember that k3OS’s root filesystem is immutable; edits made directly will be lost on reboot. Persist changes through an Ignition file that writes to /etc/containers/registries.conf.

Check the current file:

cat /etc/containers/registries.conf

A valid entry for a private registry looks like:

[ [registry \"registry.example.com\"] ]
insecure = false
username = \"myuser\"
password = \"mypassword\"

Correct any typos, ensure the double quotes are escaped properly in the Ignition config, and redeploy the node.

Step 4: Manual image pull with ctr

To isolate Kubernetes from the runtime, use the containerd CLI:

sudo ctr -n k8s image pull registry.example.com/my-image:latest

If this command succeeds but the pod still fails to start, the problem lies in the Kubernetes ImagePullSecrets or imagePullPolicy in the pod spec.

Verification and Escalation

After applying the fix (e.g., updating the Ignition file with correct credentials or correcting a network rule), verify the pod:

kubectl get pods -n <namespace> -w

If the pod reaches Running state, the issue is resolved. If the same error persists, collect the containerd journal again and consider:

• Checking for proxy environment variables that may interfere with curl or ctr.
• Confirming the registry’s TLS certificate is trusted by the node (add the CA to /etc/ssl/certs via Ignition if using a private CA).
• Opening a support ticket with the registry provider if the error is 403 despite valid credentials.