Transient service discovery inconsistencies after Consul snapshot restore

Why watchers see stale data after a snapshot restore

A Consul snapshot restore overwrites the state machine — KV store, catalog, prepared queries, sessions, and ACLs — but it does not reset the Raft log indices or the Serf gossip membership layer. The restored catalog may immediately conflict with the health-check status and node metadata that agents are still gossiping. Until the gossip protocol reconciles and health checks renew, watchers can observe missing or outdated service entries.

How long stale tombstones typically persist

Confirmed: The inconsistency window is bounded by the gossip convergence time and the health-check renewal interval. In a typical LAN cluster with default settings (serf_lan_gossip_interval = 200 ms, health-check TTL often 10–30 s), full reconciliation usually completes within a few seconds to under a minute.

Likely explanation: Tombstones (deregistration markers) linger in the Raft log until the next gossip round pushes the restored catalog to all agents and agents re-evaluate their local health state against the server. If a node’s health check has a long TTL, its stale “healthy” status can mask the restored “deregistered” state until the check expires or is manually refreshed.

Telemetry and log indicators to detect the window

• Server logs: Watch for raft and serf messages during and after restore. Lines such as memberlist: ... or catalog: deregistering service ... indicate reconciliation activity.
• Real-time event stream: Run consul monitor -type=checks -type=services immediately after restore. A burst of ServiceDeregistered followed by ServiceRegistered events signals the convergence wave.
• State comparison: Execute consul catalog nodes and consul members side by side. Discrepancies in node counts or service lists pinpoint the inconsistency window.

Configuration tunings that affect cleanup speed

• serf_lan_gossip_interval (default 200 ms) — lowering it accelerates gossip but increases CPU and network load on every agent.
• Health-check TTL / interval — shorter TTLs force faster re-evaluation of node health against the restored catalog.
• No native “force Raft log rebuild” option. The restore operation is designed to replace the state machine only; rebuilding the log would require a full cluster bootstrap, which defeats the purpose of a snapshot restore.

Caution: Aggressive gossip tuning can destabilize large clusters. Test any interval changes in staging before applying to production.

One diagnostic detail that changes the recommendation

What are your current serf_lan_gossip_interval and the typical health-check TTL/interval for the affected services? If they are already at aggressive values, further tuning yields diminishing returns and the focus should shift to operational practices (e.g., scheduling restores during low-traffic windows, using consul snapshot save with -stale for faster but potentially inconsistent backups).