Transient service discovery inconsistencies after Consul snapshot restore
0 reputation · 24 Mar 2020, 08:32 UTC
After a Consul snapshot restore, service discovery watchers sometimes return outdated or missing entries for a short period before the cluster reconciles. The goal is to understand why this transient inconsistency occurs and what factors influence its duration.
The restore operation replaces the KV store, catalog, prepared queries, sessions and ACLs but does not rebuild the Raft log. Consequently, stale tombstone entries remain in the log until the next gossip round, allowing watchers to observe mismatched state until the cluster converges.
- How long do stale tombstone entries typically persist before gossip reconciliation resolves them?
- What telemetry or log indicators can detect these temporary inconsistencies during the window?
- Are there configuration tunings that can accelerate tombstone cleanup or force a Raft log rebuild after a restore?