Diagnosing Harvester VM Startup Failures Caused by Resource Constraints

Recognizable condition

A virtual machine (VM) in Harvester stays in the Pending or Creating phase and never reaches Running. You can see this in the Harvester UI or by running kubectl get virtualmachine -n harvester-system.

Cause/diagnostic table

Possible cause	What to look for
Insufficient host CPU or memory	Nodes show low allocatable resources; VM requests exceed what is free.
Overcommitted resource quotas	Namespace or cluster quota limits are hit.
Node taints or tolerations blocking scheduling	Scheduler events indicate TaintToleration failures.
Missing KubeVirt agent	KubeVirt pods in harvester-system are not ready.
Storage class not bound	PersistentVolumeClaim remains Pending with StorageClass not found.

Ordered checks

1. Check node resource availability

   Run this command on a workstation with kubectl configured for the Harvester cluster (requires cluster‑admin or harvester‑admin role):

   kubectl describe nodes

   Look for the Allocatable and Capacity lines under CPU and Memory. If the Allocatable values are low or show 0 for the resources your VM requests, the node is exhausted.

   Risk: Misinterpreting Allocatable after a node drain can lead to unnecessary scaling.

2. Review the VM spec requests/limits

   Retrieve the VM definition (replace <vm-name> and <namespace> with your values):

   kubectl get virtualmachine <vm-name> -n <namespace> -o yaml

   Examine the domain.resources.requests block for cpu and memory. Compare these numbers to the free resources you observed in step 1.

3. Inspect Harvester/KubeVirt controller logs

   Controller logs often contain the scheduling reason. Use:

   kubectl logs -n harvester-system deployment/harvester-network-controller

   and

   kubectl logs -n harvester-system deployment/kubevirt-controller

   Search for phrases like FailedScheduling or Insufficient cpu.

4. Verify storage class and volume binding status

   Check the PVC associated with the VM’s disk:

   kubectl get pvc -n <namespace> -l kubevirt.io=domain-name,<vm-name>

   Then describe it:

   kubectl describe pvc <pvc-name> -n <namespace>

   If the Events section shows StorageClass not found or Waiting for first consumer without binding, the storage class is the issue.

5. Look for scheduling events

   Describe the VM to see events emitted by the scheduler:

   kubectl describe virtualmachine <vm-name> -n <namespace>

   In the Events table, note any FailedScheduling reasons (e.g., NodeAffinity, TaintToleration, Insufficient cpu).

Fixes tied to findings

• Add or resize nodes – If node CPU/memory is insufficient, add a new node to the cluster or increase the size of existing nodes (requires infrastructure access). After adding, run kubectl get nodes to confirm the new node appears with Ready status.
• Adjust VM resource requests – Edit the VM spec to lower requests to match available resources:

  kubectl edit virtualmachine <vm-name> -n <namespace>

  Change cpu to a value ≤ free CPU cores and memory to ≤ free memory (e.g., from 4 vCPU to 2 vCPU). Save and exit; the controller will reschedule.
• Remove conflicting taints/tolerations – If a node taint blocks the VM, either remove the taint or add a matching toleration to the VM:

  kubectl taint nodes <node-name> key=value:NoSchedule-

  or edit the VM to add tolerations: block.
• Ensure KubeVirt agent is running – Restart the KubeVirt controller pod if it is CrashLoopBackOff:

  kubectl rollout restart deployment/kubevirt-controller -n harvester-system

  Verify pods are Ready with kubectl get pods -n harvester-system.
• Bind or create appropriate storage class – If the PVC lacks a bound storage class, create one that matches your backend (e.g., Longhorn) or use the existing harvester-longhorn class:

  kubectl get storageclass

  Then patch the PVC:

  kubectl patch pvc <pvc-name> -n <namespace> -p '{"spec":{"storageClassName":"harvester-longhorn"}}'

  Warning: Changing the storage class of an existing volume may detach it; ensure you have a snapshot or backup first.
• Restart Harvester controller if needed – After fixing the above, a controller restart can clear stale state:

  kubectl rollout restart deployment/harvester-network-controller -n harvester-system

Escalation criteria

• If the VM remains Pending after applying the relevant fix and the logs show no clear scheduling error, collect a full log bundle:
• kubectl get events -n harvester-system > events.txt
• kubectl logs -n harvester-system -l app=harvester --all-containers > controller.logs
• Open a support ticket with Harvester, attaching the bundle and the VM YAML.
• Check the Harvester release notes for known issues matching your version (e.g., v1.2.0) before filing a bug.

Verification

1. After applying a fix, run kubectl get virtualmachine <vm-name> -n <namespace> and confirm the STATUS column shows Running.
2. Check node utilization to ensure you are not over‑committing: kubectl top nodes or view the Harvester dashboard; aim for CPU and memory usage below 80 % on each node.
3. Run kubectl get events -n harvester-system and verify that no FailedScheduling events persist for the VM.

Limitations

• This guide assumes Harvester ≥ 1.0 with KubeVirt enabled.
• It does not cover hardware‑level failures (e.g., CPU throttling due to overheating) or network‑related startup issues.
• Storage‑class changes are risky; always snapshot volumes before modification.