Diagnosing Persistent Volume Claim Failures in k3s with Local‑Path Provisioner

Problem & Takeaway

When a pod in a k3s cluster requests a Persistent Volume Claim (PVC) backed by the built‑in local-path storage class, you may see the PVC stuck in Pending and no PV bound. The root cause is often a missing or mis‑configured local‑path provisioner, insufficient node disk space, or stale objects. This guide walks you through a systematic check‑list, fixes tied to findings, and when to ask for escalation.

Key Concepts

• PV (Persistent Volume) – a storage resource in the cluster, pre‑provisioned or dynamically created.
• PVC (Persistent Volume Claim) – a request for storage that binds to a PV.
• StorageClass – defines the provisioner and parameters used to create PVs.
• local‑path provisioner – a daemonset that creates PVs on the node’s local disk under /var/lib/rancher/k3s/agent/data.

Step‑by‑Step Diagnostic Flow

1. Confirm the PVC is Pending

   kubectl get pvc -n <namespace> <pvc-name> -o wide
   

   Check the Status column. If it reads Pending, proceed.

2. Inspect the PVC’s Events

   kubectl describe pvc -n <namespace> <pvc-name>
   

   Look for events like Failed binding: no available PV or Failed to provision volume.

3. Verify the local‑path Provisioner DaemonSet

   kubectl -n kube-system get pods -l app.kubernetes.io/name=local-path-provisioner
   

   All pods should be Running. If any are CrashLoopBackOff or Pending, the provisioner is unhealthy.

4. Check Node Disk Capacity

   kubectl describe node <node-name> | grep -i capacity
   ssh <node-name> df -h /var/lib/rancher/k3s/agent/data
   

   Ensure the node’s disk has free space. A full disk prevents the provisioner from creating the PV file.

5. Examine Existing PVs and PVCs

   kubectl get pv,pvc -o wide | grep Pending
   

   Stale PVs can block new bindings if they reference non‑existent nodes.

6. Validate StorageClass Configuration

   kubectl get sc <storageclass-name> -o yaml
   

   Confirm provisioner: rancher.io/local-path and that volumeBindingMode is set appropriately.

7. Check Node Taints and Pod Tolerations

   kubectl describe node <node-name> | grep -i taints
   kubectl describe pod <pod-name> | grep -i tolerations
   

   Missing tolerations can prevent a pod from scheduling onto a tainted node, indirectly stalling PVC binding.

Tied Fixes for Common Findings

Finding	Fix
Provisioner pod in CrashLoopBackOff	Check logs: kubectl -n kube-system logs <pod>. Common causes: insufficient memory limits or mis‑mouted /var/lib/rancher/k3s/agent/data. Adjust limits or move the directory to a larger volume.
Node disk full	Free space by deleting unused images or logs, or expand the underlying volume. After change, restart the provisioner pod.
Stale PVs referencing missing nodes	Delete orphaned PVs: kubectl delete pv <pv-name>. Ensure the PV spec matches an existing node path.
Incorrect StorageClass or access mode mismatch	Update the PVC to use ReadWriteOnce and the correct StorageClass, or create a new StorageClass pointing to local-path.
Node taints blocking pod scheduling	Add tolerations to the pod spec: tolerations: - key: "node.kubernetes.io/unschedulable" operator: "Exists" effect: "NoSchedule".

Example Fix: Removing a Stale PV

Suppose you find a PV named pv-1234 stuck in Released state, still referencing a node that no longer exists.

# List the PV to confirm its status
kubectl get pv pv-1234 -o wide

# Delete the stale PV
kubectl delete pv pv-1234

# Verify no lingering PVCs
kubectl get pvc -o wide | grep pv-1234

After deletion, new PVCs should bind normally.

Verification Checklist

• All local‑path pods are Running.
• Node disk shows at least 10 % free space in /var/lib/rancher/k3s/agent/data.
• No orphaned PVs referencing missing nodes.
• PVC events show no binding errors.
• Pod scheduler can place pods on the node (no taint/toleration mismatch).

When to Escalate

1. Provisioner pods remain in CrashLoopBackOff after log inspection.
2. Disk expansion is not an option and the cluster is production‑critical.
3. Persistent volume claims continue to fail after all local checks.

In these cases, contact your cluster administrator or open a support ticket with the distribution vendor. Provide the PVC YAML, node disk usage, and provisioner logs.

Limitations & Best Practices

• The local-path provisioner is best suited for single‑node or low‑density clusters. In multi‑node setups, consider a networked storage solution to avoid data loss if a node fails.
• Always back up PV data before deleting stale objects.
• Monitor node disk usage proactively; set alerts for df -h thresholds.