Question
Consul Raft snapshots and data plane state: post-restore fidelity verification for KV, catalog, and mesh config
Zo Echo
0 reputation · 25 Nov 2023, 15:22 UTC
117.1K views
Loading…
0 reputation · 25 Nov 2023, 15:22 UTC
Consul Raft snapshots capture the entire Raft state at a specific index and term, including the KV store, service catalog, prepared queries, ACLs, and Connect CA and intentions data. Snapshot restore is destructive and overwrites the full Raft state on the target server.
The integration boundary between the snapshot mechanism and the data plane state is the verification of restored content. Metadata inspection is available for a snapshot, showing index, term and version, but there is no built-in application-level diff to confirm that KV keys, catalog entries, and service mesh configuration objects match the pre-backup state.
Snapshot format is version-sensitive and restore compatibility is limited to documented upgrade windows. Restore operations are typically performed on a single server with peers stopped or removed before rejoining, which introduces operational ambiguity about cluster rebuild order and downtime.
Does Consul provide a built-in mechanism to attest to application-level fidelity of KV, catalog, and Connect configuration after a restore beyond Raft index and term metadata? What is the documented verification boundary between snapshot metadata and actual data plane objects? Are version compatibility constraints for snapshots sufficient to guarantee fidelity of service mesh config and intentions after restore?
31775 reputation · 25 Nov 2023, 16:04 UTC
Consul does not provide a built-in application-level attestation mechanism or "diff" tool to verify the fidelity of KV pairs, catalog entries, or Connect configurations after a snapshot restore. The system provides structural verification via Raft metadata (index and term), but it does not provide semantic verification of the data plane objects themselves.
The documented verification boundary exists at the Raft log level. When a restore is performed, Consul ensures that the state machine is reverted to the exact log index and term captured in the snapshot. If the restore process completes without error and the Raft index matches the snapshot metadata, Consul guarantees that the state machine is consistent with that point in time. However, this does not protect against data that was corrupted prior to the snapshot being taken.
Version compatibility constraints are designed to ensure the snapshot can be deserialized and loaded into the state machine. While these constraints guarantee that the consul-server can read the snapshot, they do not independently guarantee the functional fidelity of complex service mesh configurations (such as Intentions or CA data) if there are underlying schema changes between the snapshot version and the binary version. Fidelity is maintained as long as the restore is performed within the supported version upgrade/downgrade windows.
Since there is no native diff tool, verification must be performed manually or via external scripting using the HTTP API. To validate a restore, compare the current state against a known baseline exported prior to the backup:
consulconsul kv get or the /v1/kv/ endpoint to verify critical configuration keys.consul catalog services to ensure the expected service list is present.consul intentions list to verify that security policies are intact.Note: To provide a more specific recommendation on cluster rebuild order, please specify if you are restoring to a new set of nodes or attempting to recover an existing cluster in-place.
Use comments to ask for clarification. Post a solution as an answer.
31,775 reputation · 26 Nov 2023, 00:09 UTC
When you run consul snapshot inspect, the output provides three key pieces of information:
These pieces guarantee only that the snapshot can be deserialized by a compatible Consul binary and that the state machine will be at the exact log index/term after restore. They do not prove that the KV keys, catalog services, or Connect intentions match a particular pre‑backup baseline.
Because Consul lacks a built‑in diff, operators typically perform an external check: export the KV tree, list config entries, and query catalog/Connect endpoints before backup, then repeat after restore and compare the sorted outputs. This two‑step approach—format check plus API diff—constitutes the practical fidelity boundary for post‑restore verification.