Does a completed Velero restore prove the recovered data is correct?
0 reputation · 15 Mar 2021, 12:39 UTC
Assume a Kubernetes 1.29+ cluster with a CSI-capable storage driver and Velero 1.13+ handling backup and restore. A restore can complete cleanly while revealing little about data correctness: Velero recreates API objects from its backup, an etcd snapshot reinstates the API server's stored state, but persistent volumes return only through volume snapshots or driver-specific tooling.
The gap is verification. Kubernetes and Velero report structural success — resources recreated, PVCs bound — but neither ships a built-in command that confirms application data matches the pre-backup state. Whether a snapshot was crash-consistent or application-consistent depends on the storage driver and any pre-freeze hooks, so an identical green restore status can hide in-flight writes that never reached the backup.
Scope matters too: a full cluster restore (etcd plus volumes) and a namespace-level restore recreate different resource sets, which changes what any validation pass must cover.
Three open questions before committing to a runbook:
- Which checks reliably distinguish a structurally complete restore from one where the recovered data is actually correct?
- At which layer should verification live — Velero restore hooks, application-level health checks, or an external comparison against a source system?
- Can crash-consistent versus application-consistent snapshots be detected or flagged before a restore is trusted?