Harvester's Mandatory Longhorn Integration: Architecture, Boundaries, and Operational Checks

Problem: Understanding Harvester's Storage Dependency

When evaluating Harvester for virtual machine workloads, a common question is why the platform ships with Longhorn enabled by default and does not allow alternative CSI drivers without explicit, unsupported configuration. The answer lies in the architectural requirements of KubeVirt's VM disk APIs and the operational guarantees Harvester aims to provide.

Requirements Driving the Design

Harvester must satisfy three core requirements for VM disks:

1. ReadWriteOnce/ReadWriteMany block storage - KubeVirt's ContainerDisk and DataVolume APIs need a storage class that can expose raw block devices with snapshot and clone capabilities.
2. Live migration and snapshot/restore - These features require at least one healthy replica on a different node to maintain volume availability during node evacuations.
3. Integrated management - Harvester's UI and CLI rely on storage-specific CRDs e.g., Volume, Backup to provide a single pane of glass for disk health, snapshots, and backups.

Longhorn provides RWX/RWO block storage with native snapshot/clone semantics on bare metal without needing an external CSI driver, fulfilling all three requirements.

Smallest Suitable Design

Even a single-node Harvester deployment must run Longhorn with a replica count of at least one. However, because live migration and snapshot/restore depend on having a replica on a different node, the default replica count is three. The design can be reduced to one replica for edge use cases, but this explicitly disables HA-related features:

• VM live migration is blocked.
• Volume snapshots and clones are unavailable.
• Disaster-recovery workflows that rely on replica symmetry cannot be used.

Harvester's harvesterchartconfig allows setting spec.longhorn.replicaCount=1, but the documentation marks this configuration as not a supported HA configuration.

Trust and Data Boundaries

Harvester enforces strict namespace-level isolation to prevent user workloads from bypassing its storage abstraction:

• Control plane - Runs in harvester-system with a cluster-admin service account. It manages the Harvester CRDs and the Longhorn operator.
• Longhorn system - Operates in longhorn-system with its own service account. It owns the storage engine, replica pods, and the longhorn-manager.
• User VM namespaces - VMs created via KubeVirt run in tenant namespaces with PodSecurity Admission PSA restrictions v1.4+ or PodSecurityPolicy PSP v1.3. These policies prevent direct access to Longhorn's API or CRDs, ensuring that all storage interactions go through the Harvester controller.

This separation creates a trust boundary: the control plane can provision and monitor Longhorn volumes, but user workloads cannot manipulate replica settings or snapshot policies directly.

Operational Checks

Harvester exposes several endpoints that operators can query to verify storage health:

• Node-disk health - GET /apis/harvesterhci.io/v1beta1/nodes/{node}/disks returns disk conditions reported by the harvester-node-disk-manager controller.
• Longhorn replica readiness - kubectl -n longhorn-system get volume <pvc-name> -o jsonpath='{.status.robustness}' should show Healthy.
• Replica rebuild detection - If Longhorn reports a replica rebuild exceeding 30 minutes or three consecutive I/O errors, the node-disk manager marks the disk Unavailable.

Example verification command run with cluster-admin rights:

# Check Longhorn integration status via HarvesterChartConfig
kubectl -n harvester-system get harvesterchartconfig -o yaml | grep -A2 longhorn

The output should contain enabled: true under the longhorn block if the default installation is intact.

Failure Modes and Mitigations

Longhorn Manager OOM

The longhorn-manager pod is given a memory limit of 500 MiB and a system-cluster-critical priority class. If it is OOMKilled:

• The instance-manager sidecar on each node continues serving existing replica I/O, so active VM disks remain accessible.
• New volume operations e.g., creating a snapshot are queued until the manager restarts, which typically occurs within 60 seconds.

Network Partition Between Nodes

Longhorn uses gRPC for replica synchronization. When a node loses quorum fewer than two healthy replicas out of three:

• The affected volume enters a Degraded state and blocks new writes.
• Harvester's VM controller watches VolumeAttachment conditions. If a target node holds a healthy replica, it attempts live migration of the VMI; otherwise it pauses the VMI until connectivity is restored.

MTU Mismatch

Longhorn replica sync fails if the underlying network MTU is below 1500 bytes. The Harvester installer validates MTU >= 1500 at deploy time but does not re-validate after installation. Operators should monitor for replica Failed conditions and verify MTU on affected nodes.

Conditions That Would Change the Design

Harvester's tight coupling to Longhorn can be loosened only by disabling the embedded storage:

# In HarvesterChartConfig, set:
spec:
  longhorn:
    enabled: false

When longhorn.enabled=false:

• Users must provide an external CSI driver that supplies RWX/RWO block storage with snapshot/clone support e.g., Ceph RBD via CSI, vSphere CSI.
• Harvester's integrated VM snapshot/backup UI, single-pane monitoring, and automated disk provisioning are lost; these features are marked as unsupported in the SUSE support matrix.
• Velero-based backup/restore only works with Longhorn snapshots; external CSI volumes are silently excluded from backup jobs.

Additionally, Harvester pins a specific Longhorn version per release e.g., Harvester 1.3.2 -> Longhorn 1.6.2. Skipping Longhorn minor versions during a Harvester upgrade is blocked by the harvester-upgrade controller; forcing a Longhorn upgrade first requires manual CRD conversion.

Limitations and Practical Verification

Operators should keep the following limits in mind:

• The default 3-replica policy triples raw disk consumption; Harvester does not auto-expand underlying disks.
• Single-node clusters with replicaCount=1 lose live migration, snapshots, and DR capabilities - not a supported HA configuration.
• Disk encryption LUKS is performed at the Longhorn replica level; key rotation requires the longhornctl CLI and VM downtime.
• Harvester 1.4+ uses PodSecurity Admission; custom PSPs for Longhorn volumes must be migrated to valid PSA labels to avoid breakage.

A practical way to confirm that Longhorn remains the active storage backend after any configuration change is to check the storage class used by a newly created VM disk:

# Create a test PVC and inspect its storage class
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 5Gi
EOF

kubectl get pvc test-pvc -o jsonpath='{.spec.storageClassName}'

If the output shows longhorn, the default integration is still in effect. Any other storage class indicates an external CSI driver is being used.

Takeaway

Harvester mandates Longhorn because it is the only storage solution that natively supplies the block-level, snapshot-capable RWX/RWO volumes required by KubeVirt's VM disk APIs while staying within Harvester's trust boundaries. The smallest viable design still runs three replicas to enable live migration and snapshots; reducing to a single replica disables those features and is not a supported HA configuration. Operators can verify integration status via the HarvesterChartConfig and Longhorn volume health checks, and they should be aware of the failure-mode mitigations manager OOM, network partition, MTU and the consequences of disabling Longhorn loss of integrated UI, backup, and official support.