Question
Limits of Consul Snapshot: Agent‑Level Data Exclusion and Restoration Options
Nia Moss
0 reputation · 27 Dec 2025, 02:10 UTC
71.8K views1
Loading…
0 reputation · 27 Dec 2025, 02:10 UTC
Ensure that a Consul cluster can be fully recovered, including both server‑side state and agent‑specific data such as local service definitions and intentions, after a disaster.
The built‑in consul snapshot command captures a consistent copy of the server raft state but deliberately omits agent‑level data. Administrators must decide whether to supplement snapshots with a separate backup mechanism for agent data, and how to verify that the supplemental data is correctly restored.
Questions remain about the supported way to back up agent data, the impact of version differences on that data, and the verification steps needed after a combined restore.
35275 reputation · 27 Dec 2025, 12:35 UTC
The built-in consul snapshot command captures only server-side Raft consensus state. It does not include agent-local service definitions, intentions, or other client-agent configuration. To recover fully, you must supplement snapshots with a separate agent-data backup and verify both layers independently.
Confirmed behavior: consul snapshot save serializes the server state machine, not the agent process's local catalog or configuration files. Agent-local registrations are typically held in memory or in the agent's data directory and are lost if the agent host is wiped.
Likely explanation: Because agents re-register services on restart from their configuration files, the practical impact of losing agent data depends on whether services are defined via static files, the HTTP API, or remote orchestration. If definitions exist only in agent memory, a snapshot restore alone will not bring them back.
consulconsul snapshot save for Raft state and consul snapshot restore for recovery.consul export to produce structured JSON of live agent state. The -service and -intention flags cover the most common agent-level objects./etc/consul.d) alongside the agent data directory. File-based backups are version-sensitive; confirm compatibility before restoring onto a different Consul minor version.Agent data exported via consul export is generally usable within the same Consul minor release. Restoring exported JSON across major versions, or from a newer minor version to an older one, may require manual schema adjustment. Treat cross-version agent-data restores as a change that needs testing in a non-production environment first.
After restoring the server snapshot and restarting agents, verify both layers:
consul snapshot verify to confirm the restored server archive is intact.consul export -service -intention -out /dev/stdout before backup and again after restore, then diff the output.consul catalog services and consul catalog nodes.consul intention list and confirm the set matches expectations, allowing for ACL propagation delays.One detail changes the recommendation: are your service definitions and intentions registered statically via agent configuration files, or dynamically via the HTTP API / consul services register? If they are static, file-based backup may be sufficient and simpler. If they are dynamic, consul export is the more reliable path because there may be no on-disk source of truth.
Note: Consul's agent data model and export format have evolved across releases. Before relying on cross-version restore, validate the exact version pair in your environment.
Use comments to ask for clarification. Post a solution as an answer.
6,210 reputation · 27 Dec 2025, 05:01 UTC
consul snapshot save captures server Raft state—catalog, checks, key/value entries, and Connect intentions—but not agent-only configuration files or services defined solely in an agent's local config directory. Those agent-local definitions are invisible to the snapshot and will not reappear after consul snapshot restore.
A practical restore sequence is: restore the server snapshot on an empty or recovered cluster, redeploy agent configuration from configuration management, run consul validate against the target agent version, then reload or restart each agent. Verify with consul catalog services, consul intention list, and health-check output against a pre-disaster baseline. Avoid restoring a snapshot from a newer Consul minor release into an older cluster without staging validation.