Snapshot consistency limits on active VMs with Ceph-backed volumes

Question

Snapshot consistency limits on active VMs with Ceph-backed volumes
Harvester’s snapshot API allows administrators to capture a VM’s disk state for backup and recovery. The API is documented for both local and Ceph‑backed volumes, but the guarantees around block‑level consistency when the VM is actively writing are not explicitly addressed. The retention policy is configurable, yet its impact on storage quota and concurrent snapshot performance remains undocumented. This uncertainty hampers planning for large‑scale snapshot operations, especially when multiple VMs share Ceph storage.
Key constraints include active I/O during snapshot, the choice of storage backend, and the configured retention window. The main uncertainty is whether snapshots preserve a consistent block state on Ceph and how retention interacts with storage limits.
Specific questions:

Does the /snapshots endpoint guarantee block‑level consistency for Ceph‑backed volumes when a VM is actively writing?
How does the retention policy affect storage quota and performance during concurrent snapshot operations on Ceph?
What safeguards, if any, does Harvester provide when a VM with pending snapshots is live‑migrated?

Tasadduq Burney · Answer

Direct Answers
Block-level consistency: No. The /snapshots endpoint creates a crash-consistent point-in-time snapshot via the CSI VolumeSnapshot API. Active guest writes are not guaranteed to be flushed; the on-disk state matches what you would see after an abrupt power loss. Application-consistent snapshots require the Harvester guest agent or a manual fsfreeze inside the VM before triggering the snapshot.
Retention policy impact: The policy deletes the oldest VolumeSnapshot objects once the configured count is exceeded. Each retained snapshot consumes additional Ceph RBD capacity proportional to the unique changed blocks since its parent. Quota usage grows linearly with the number of retained snapshots and the write rate between them. Concurrent snapshot creation or deletion adds load on the Ceph cluster (metadata operations, object writes, and potential OSD contention).
Live-migration safeguard: While a VolumeSnapshot is in Pending or Creating state, Harvester/KubeVirt locks the underlying PVC. Any virtctl migrate attempt for that VM will fail or time out until the snapshot completes or is cancelled. No automatic quiesce or migration coordination occurs.

Why This Matters
Crash consistency is sufficient for many backup workflows (databases with WAL replay, stateless workloads). For databases or applications that require a clean on-disk state, you must integrate the guest agent or script fsfreeze/fsunfreeze around the snapshot API call. The retention policy is a simple count-based cleanup; it does not throttle snapshot creation or enforce storage quotas beyond Kubernetes PVC limits.

Assumptions & Version Context

Harvester ≥ 1.2 with the Ceph CSI driver (rbd.csi.ceph.com).
KubeVirt handles PVC locking during snapshot operations.
Behavior may differ on older Harvester releases or with non-Ceph CSI drivers.

Verification Steps (Run in a Test Cluster)

Deploy a test VM with a Ceph RBD volume and enable the Harvester guest agent.
Start a steady write workload (e.g., fio --rw=randwrite --size=1G --ioengine=libaio).
Take a snapshot via the Harvester API without quiescing.
Compare the snapshot data to a quiesced snapshot using rbd diff or qemu-img check to detect missing writes.
While a snapshot is Creating, run virtctl migrate <vm> and confirm it is blocked.
Create concurrent snapshots on multiple VMs sharing the same Ceph pool; monitor ceph df, rbd snap ls, and ceph perf top to observe quota growth and IOPS impact.

One Missing Detail
What is your Harvester version and Ceph CSI driver version? The locking semantics and guest-agent integration changed between 1.1 and 1.2, so the exact behavior depends on that combination.

Snapshot consistency limits on active VMs with Ceph-backed volumes