Leveraging Rook‑Ceph RBD for Harvester VM Disks: A Practical Guide

Why RBD for Harvester VMs?

Running virtual machines on a Kubernetes‑based hypervisor like Harvester requires persistent block storage that is both highly available and scalable across nodes. Local disks are simple but lack redundancy, while network‑attached storage introduces latency. Rook‑Ceph’s RBD (RADOS Block Device) offers a distributed block layer that can be dynamically provisioned, snapshot‑ed, and replicated—all via the CSI interface that Harvester already consumes for pod storage.

Feature Overview

Harvester 1.0+ automatically creates a harvester‑rbd StorageClass once a Rook‑Ceph cluster is present in the same namespace. The class points to the RBD CSI driver shipped with Rook 1.8+. When a VM definition references this class, Harvester provisions a Ceph image, attaches it through KubeVirt’s virt‑console, and manages it like any other block device.

Deploying the RBD Backend

1. Install Rook‑Ceph
   

   # helm install rook-ceph rook-ceph/rook-ceph --namespace rook-ceph --create-namespace
   

2. Verify the RBD CSI driver is available
   

   # kubectl get pods -n rook-ceph | grep rbd-csi
   

3. Deploy Harvester (assumes the same namespace harvester-system)
   

   # helm install harvester harvester/harvester --namespace harvester-system
   

4. Check the automatically created StorageClass
   

   # kubectl get storageclass
   NAME              PROVISIONER               RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
   harvester-rbd    rook.io/rbd                Delete          WaitForFirstConsumer   false                   5m
   

Provisioning a VM with RBD

Below is a minimal VM YAML that requests a 20 Gi RBD disk. Replace my-namespace with your own.

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  name: rbd-demo
  namespace: my-namespace
spec:
  running: true
  template:
    spec:
      domain:
        devices:
          disks:
          - name: rootdisk
            disk: {}
        resources:
          requests:
            memory: 4Gi
      volumes:
      - name: rootdisk
        persistentVolumeClaim:
          claimName: rbd-demo-root
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-demo-root
  namespace: my-namespace
spec:
  storageClassName: harvester-rbd
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

Apply the file:

# kubectl apply -f rbd-demo.yaml

Once the VM is running, verify the underlying RBD image exists:

# ceph osd pool ls
rbd
# ceph df
... (pool usage details)

And check the PVC status:

# kubectl describe pvc rbd-demo-root -n my-namespace
Name:          rbd-demo-root
Namespace:    my-namespace
StorageClass: harvester-rbd
Volume:       rbd-demo-root
Capacity:     20Gi
Access Modes: RWO
Phase:        Bound

Trade‑offs and Limitations

• Latency vs. Scalability – RBD introduces network I/O. For latency‑sensitive workloads, local SSDs may still be preferable.
• Resizing Unsupported – Harvester currently does not support expanding an RBD disk. To grow a VM’s storage, delete the VM and recreate it with a larger PVC.
• Ceph Pool Capacity – If the rbd pool is full, provisioning fails. Monitor ceph df and add OSDs as needed.
• Monitor Stability – A Ceph monitor outage can block new disk creation. Ensure at least three monitors for high availability.
• Snapshot Overhead – VM snapshots via RBD may take several seconds and can temporarily block I/O on the image.

Next Steps

1. Enable Replication – Configure Rook’s Ceph replication rules to spread data across nodes.

2. Automate Backup – Use Harvester’s snapshot API to capture VM state, then store snapshots in Ceph’s snapshot list.

3. Monitor Performance – Deploy Prometheus exporters for Ceph and KubeVirt to track IOPS, latency, and queue depth.

4. Plan for Scaling – As VM count grows, add OSDs and monitor pool utilization to avoid out‑of‑space errors.

Conclusion

Integrating Rook‑Ceph RBD as the storage backend for Harvester VMs gives you a distributed, resilient block layer with minimal configuration. The automatic harvester‑rbd StorageClass, CSI compatibility, and snapshot/clone features make it a compelling choice for most workloads. Just be mindful of the latency trade‑off and the current lack of dynamic resizing support.