Harvester Architecture Note: Integrated KubeVirt and Longhorn Design

Problem and Takeaway

When deploying Harvester you need a clear view of how its control plane, KubeVirt virtual machines, and Longhorn storage interact, where trust boundaries lie, and what operational checks keep the system healthy. This note outlines the minimal viable architecture, the failure modes you should monitor, and the conditions that would force a redesign.

Requirements

Harvester (v1.2+) assumes a Kubernetes cluster that provides:

• At least three nodes for high‑availability (HA) etcd and control‑plane services.
• Each node contributes CPU, memory, and a local SSD/NVMe device that Longhorn will use as a storage backend.
• Network connectivity that allows the default CNI (Canal or Multus) to reach all pods.

Harvester installs the KubeVirt operator and Longhorn as system components via Helm charts bundled with the distribution.

Smallest Suitable Design

The minimal viable layout is a three‑node control‑plane where every node runs:

• Rancher/Harvester management API.
• Kubernetes API server and etcd (forming an HA cluster).
• KubeVirt components: virt‑operator, virt‑controller, virt‑handler.
• Longhorn manager and the Longhorn engine/replica pods that consume the local disks.

Virtual machines are defined as KubeVirt VirtualMachine custom resources. Their disks are represented by Longhorn volumes, which provide persistent block storage without an external SAN. Workload pods (the virt‑launcher containers) run on any node, accessing their Longhorn volume via the CSI driver.

Trust and Data Boundaries

The management plane trusts only authenticated Rancher/Harvester admin users (via the built‑in auth proxy or external OIDC). All VM workloads execute inside KubeVirt pods that are confined by SELinux/AppArmor profiles and run as non‑root users. Longhorn stores data encrypted at rest (if the encryption feature is enabled) and replicates each volume across at least two nodes, keeping tenant VM data separate from the control‑plane etcd store.

Operational Checks

Harvester exposes health‑check endpoints for the KubeVirt virt‑handler and Longhorn manager. Administrators can:

• Run kubectl get pods -n kubevirt to verify virt‑operator, virt‑controller, and virt‑handler are Running.
• Execute kubectl get pods -n longhorn and confirm manager, engine, and replica pods report Healthy.
• Use kubectl get vmi to inspect the status of virtual machine instances.
• Check Longhorn volume health with the CLI: longhorn-cli volume get .
• Validate node readiness via harvesterctl node list (or kubectl get nodes).

Failure Modes

Storage node loss: Longhorn detects the missing replica and initiates a rebuild using the remaining healthy nodes. If fewer than two replicas stay online, the volume goes offline and any attached VM is paused. Harvester will taint the affected node to prevent new workloads from landing there.

Control‑plane node loss: The HA etcd cluster tolerates a single node failure. Losing two control‑plane nodes breaks quorum; manual etcd recovery is required before the Harvester API can serve requests.

Version upgrade incompatibility: Upgrading KubeVirt or Longhorn may introduce API changes. Harvester recommends draining VMs, upgrading to a compatible Harvester release, then verifying the new component versions with the checks above.

Conditions That Trigger a Redesign

You should revisit the architecture when:

• Storage demand exceeds 80 % of the combined local SSD/NVMe capacity across nodes, risking volume offline events.
• CPU heterogeneity blocks live migration (e.g., mixing Intel Skylake with Ice Lake). In such cases, either set CPU affinity per VM or disable migration and accept downtime for node maintenance.
• Network isolation requirements change (e.g., need for strict SR‑IOV or VLAN‑tagged secondary networks) and the current CNI cannot satisfy them without misconfiguration.

Example Configuration and Verification

Below is a minimal VirtualMachine manifest that uses a Longhorn‑backed disk. Apply it after confirming the prerequisites.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: test-vm
spec:
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/vm: test-vm
    spec:
      domain:
        resources:
          requests:
            memory: 512Mi
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: longhorn-disk
              disk:
                bus: virtio
      volumes:
        - name: containerdisk
          containerDisk:
            image: kubevirt/fedora-cloud-container-disk-demo:latest
        - name: longhorn-disk
          persistentVolumeClaim:
            claimName: test-vm-longhorn-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-vm-longhorn-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 10Gi

To verify the end‑to‑end workflow:

1. Create the PVC and VM: kubectl apply -f vm-example.yaml.
2. Start the VM: kubectl start vm test-vm.
3. Check that the virt‑launcher pod is running and the Longhorn volume is bound: kubectl get pvc test-vm-longhorn-pvc -o wide.
4. Access the console: kubectl console vm test-vm (or use virsh via the virt‑controller).
5. Inside the VM, confirm the disk is present: lsblk should show a /dev/vdb matching the 10 GiB size.

Limitations and Practical Checks

Harvester’s integrated Longhorn assumes sufficient local disk capacity. Monitor pool usage with longhorn-cli get settings and ensure the storage-over-provisioning-percentage stays below 80 %. If usage approaches this threshold, either add more storage nodes or delete unused volumes.

Live migration depends on compatible CPU models across nodes. Run kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.cpuModel}' to verify uniformity; if heterogeneous, set cpuModel: host-passthrough in the VM spec or disable migration.

Network isolation for VMs relies on the underlying CNI. After any node upgrade, run kubectl get pods -n to confirm CNI pods are healthy, and test VM connectivity with a simple ping or curl to a known service.