Diagnosing Node Not Ready and Scheduling Disabled in Harvester HCI

Problem Statement

In a Harvester HCI cluster, a worker node may transition to a NotReady state or receive a scheduling disabled taint. These conditions prevent the placement of new Virtual Machines (VMs) and can lead to workload instability. Because Harvester integrates KubeVirt for virtualization and Longhorn for distributed storage, these failures typically originate from a breakdown in the heartbeat between the node and the control plane, a failure in the Longhorn storage engine, or a resource profile mismatch.

Diagnostic Matrix

Recognizable Condition	Likely Cause	Primary Diagnostic Tool
Taint: node.kubernetes.io/unreachable	Network partition or kubelet crash	kubectl describe node
Pod stuck in ContainerCreating	CSI timeout or Longhorn volume attachment failure	kubectl get events
Node NotReady but SSH is active	Longhorn engine failure or API server latency	Longhorn UI / journalctl
Scheduling Disabled (No Taint)	ResourceProfile quota exceeded	kubectl get node -o jsonpath

Ordered Diagnostic Checks

1. Validate Node Connectivity and Taints

Check the current status of the cluster members to identify if the issue is isolated to one node or systemic.

# Run on any control-plane node with cluster-admin permissions
kubectl get nodes -o wide
kubectl describe node <node-name>

Look for the Conditions section. If Ready is False or Unknown, check for the unreachable taint, which indicates the control plane has lost contact with the node's kubelet.

2. Inspect Longhorn Storage Health

Harvester nodes rely on the Longhorn engine to mount volumes. If the engine is crashing, the node may be marked NotReady.

# Check for running engine pods on the specific node
kubectl -n longhorn-system get pods -l app.kubernetes.io/name=longhorn-engine -o wide | grep <node-name>

If the pod is in CrashLoopBackOff, inspect the logs for disk pool errors:

kubectl logs -n longhorn-system <engine-pod-name> --tail=100

3. Analyze Kubelet Heartbeats

If the network is up but the node is NotReady, the kubelet may be failing to send heartbeats due to local resource pressure.

# Run directly on the affected worker node as root
sudo journalctl -u kubelet -n 200 --no-pager

Search for "heartbeat" or "timeout" errors. If the logs show API server timeouts, check the control plane for memory pressure.

4. Verify Resource Profile Quotas

Harvester uses a ResourceProfile to define the maximum CPU, memory, and disk space a node can utilize. If these limits are hit, scheduling is disabled.

# Check the assigned profile label
kubectl get node <node-name> -o jsonpath='{.metadata.labels.harvesterhci\.io/resource-profile}'

Compare the profile's maxDiskSpace against the actual disk usage in the Longhorn UI. If the usage exceeds the profile limit, the node will stop accepting new workloads.

5. Review CSI Volume Events

If the node is Ready but VMs cannot start, check for Container Storage Interface (CSI) failures.

# Filter events for volume attachment errors
kubectl get events --field-selector involvedObject.kind=Pod -o wide | grep -i "FailedAttach"

FailedAttachVolume errors typically indicate that the Longhorn replica is unavailable or the volume is still attached to a previous node (stuck detach).

Fixes Based on Findings

• For Network Partitions: Restore connectivity to the control plane on port 6443. If the kubelet is hung, restart it: systemctl restart kubelet.
• For Longhorn Engine Crashes: Verify the local disk pool is not 100% full. If the disk is healthy, delete the crashing engine pod to force a restart.
• For Resource Profile Mismatches: Update the ResourceProfile object to increase maxDiskSpace or maxCPU to match the physical hardware capacity.
• For CSI Timeouts: In the Longhorn UI, check for "orphaned" volumes. Manually detach the volume from the old node before attempting to start the VM on the new node.

Rollback and Verification

If you modified a ResourceProfile or restarted the kubelet, verify the result using:

kubectl get nodes

The node should transition to Ready within 60 seconds. If you manually detached a volume in Longhorn, ensure the VM pod moves from ContainerCreating to Running.

Limitations and Precautions

• Data Loss Risk: Never delete Longhorn volumes via the CLI unless you have verified that enough healthy replicas exist elsewhere in the cluster.
• Network Disruption: Avoid modifying bridge or VXLAN configurations on the node while the cluster is production-active, as this can disrupt VM-to-VM traffic.
• Version Parity: Ensure KubeVirt and Harvester versions are compatible before applying updates to resolve scheduling bugs.

Escalation Criteria

Escalate to senior engineering or vendor support if:

• The node remains NotReady after a kubelet restart and network validation.
• CSI FailedAttach errors persist after manually clearing volume attachments.
• Control plane API latency remains above 5 seconds consistently, causing cluster-wide heartbeat failures.