Dynamic Storage in Rancher: Longhorn, Rook Ceph, or HostPath – A Decision Guide

Why the Decision Matters

When you run workloads on a Rancher‑managed Kubernetes cluster, the way you provision persistent storage can affect portability, resilience, and operational overhead. Kubernetes supports dynamic provisioning through the Container Storage Interface (CSI) – a standardized plugin model that lets a StorageClass automatically create a PersistentVolume (PV) when a PersistentVolumeClaim (PVC) is requested. Choosing the right CSI driver (or deciding to use a simpler hostPath PV) can make or break your cluster’s reliability and ease of migration.

Decision Context and Constraints

• Portability: Workloads may move between clusters or cloud providers.
• Scalability: Number of nodes, storage capacity, and performance expectations.
• Operational Overhead: How many operator pods, configuration files, and monitoring tasks are acceptable.
• Data Resilience: Need for replication, snapshots, or backups.
• Use‑Case: Development/testing vs production workloads.

Supported Options

Option	CSI Driver / Mechanism	Pros	Cons	Typical Use‑Case
Longhorn	Lightweight CSI with built‑in replication	Easy install via Rancher catalog, automatic snapshots/backup, < 3‑node fault tolerance	Limited raw performance, not suitable for very large clusters	Small‑to‑medium clusters needing resilient block storage
Rook Ceph	Operator that deploys Ceph cluster as CSI	Scalable block, object, and file storage; high performance	Complex installation, higher resource consumption, requires Ceph knowledge	Large, multi‑tenant environments needing high capacity
HostPath Local PV	Node‑local storage via hostPath volumes	Zero operator overhead, trivial to set up	Data tied to node, no replication or snapshots, not portable	Ephemeral dev/testing workloads

Trade‑Off Summary

• Portability: Longhorn and Rook Ceph expose the same PVC API across clusters, enabling identical YAML files. HostPath PVs break this abstraction.
• Scalability: Rook Ceph scales to petabytes; Longhorn scales to tens of terabytes but is limited by the number of replicas. HostPath is inherently limited by node capacity.
• Operational Footprint: Longhorn runs ~10 pods per node; Rook Ceph runs dozens of operator and monitor pods. HostPath has no extra pods.
• Resilience: Longhorn provides 3‑node replication; Rook Ceph offers tunable replication and erasure coding. HostPath offers none.
• Performance: Rook Ceph can deliver SSD‑grade throughput; Longhorn is suitable for moderate workloads; HostPath is limited by the host’s I/O path.

Concrete Longhorn Implementation

Below is a step‑by‑step guide to installing Longhorn via Rancher’s catalog, creating a StorageClass, provisioning a PVC, and validating data resilience with a node drain.

1. Install Longhorn from the Catalog

# In the Rancher UI, navigate to Resources > Catalog
# Search for "Longhorn" and click Install
# Use the default values (e.g., namespace: longhorn-system)
# Wait until all pods in longhorn-system are Running

Verification: In the Rancher UI, go to Workloads > longhorn-system and confirm that longhorn-manager, longhorn-driver-deployer and the longhorn-* engine pods are in the Running state.

2. Create a Longhorn StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn
provisioner: driver.longhorn.io
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
reclaimPolicy: Delete
volumeBindingMode: Immediate

Apply with kubectl apply -f longhorn-sc.yaml. The numberOfReplicas parameter is optional; Longhorn defaults to 3 for fault tolerance.

3. Request a PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-pvc
spec:
  storageClassName: longhorn
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Apply with kubectl apply -f demo-pvc.yaml. After a few seconds, kubectl get pvc demo-pvc should show Bound and a PV name that starts with pvc-.

4. Deploy a Test Pod that Uses the PVC

apiVersion: v1
kind: Pod
metadata:
  name: demo-app
spec:
  containers:
    - name: busybox
      image: busybox
      command: ["sleep", "3600"]
      volumeMounts:
        - mountPath: "/data"
          name: data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: demo-pvc

Apply with kubectl apply -f demo-pod.yaml. Verify that the pod is Running and that it can write to /data by exec-ing into the container and creating a file.

5. Validate Resilience: Node Drain Test

1. Identify one of the Longhorn replica nodes: kubectl get nodes -l longhorn.io/replica=true -o name.
2. Drain that node: kubectl drain <node-name> --ignore-daemonsets --delete-local-data. Longhorn will automatically evict the replica pods to other nodes.
3. Confirm the pod demo-app is rescheduled on a different node: kubectl get pod demo-app -o wide.
4. Exec into the pod and read the file you created earlier; data should persist.
5. Uncordon the node: kubectl uncordon <node-name>.

If the file remains intact, the replication worked as expected.

6. Snapshot and Restore Workflow

1. In the Rancher UI, go to Storage > Longhorn > Volumes > demo-pvc and click Take Snapshot. Note the snapshot name.
2. Delete the PVC: kubectl delete pvc demo-pvc. The underlying PV and data are preserved in the snapshot.
3. Create a new PVC from the snapshot:

   apiVersion: longhorn.io/v1beta1
   kind: VolumeSnapshotContent
   metadata:
     name: demo-snap
   spec:
     driver: driver.longhorn.io
     source:
       name: <snapshot-name>
     volumeHandle: <volume-handle>
     deleted: false
   

   (Replace placeholders with actual values from the UI.)
4. Verify that the new PVC binds and that the pod can read the snapshot data.

Limitations and Practical Checks

• Longhorn requires at least three nodes for full fault tolerance. In a two‑node cluster, replication cannot survive a node failure.
• Monitor CPU and memory usage of Longhorn pods; they can grow with the number of replicas.
• Ensure node affinity labels (topology.kubernetes.io/zone) are set if you want to spread replicas across zones.
• Always test backup/restore in a staging environment before relying on it in production.

When to Choose HostPath

HostPath PVs are acceptable for quick dev prototypes or stateless services where data loss is tolerable. They require no CSI operator and can be defined inline:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: hostpath-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
    type: DirectoryOrCreate

However, remember that the data is bound to the node’s local disk; draining or upgrading the node will delete the data.

Conclusion

For most production workloads on Rancher, Longhorn offers the sweet spot of dynamic provisioning, replication, and ease of use. If you need enterprise‑grade scalability and can afford the operational overhead, Rook Ceph is the next step. Reserve HostPath for non‑critical, temporary workloads where simplicity outweighs resilience.