Prepared query results inconsistent after Consul snapshot restore

Direct Answer

Consul does not maintain a separate in-memory index for prepared queries that requires rebuilding after a snapshot restore. Prepared queries are stored as objects in the servers' Raft-replicated state store (memdb) and are evaluated against that store at execution time. A successful snapshot restore replaces the entire Raft state, so prepared query definitions and the catalog data they query are immediately consistent with the snapshot contents.

Why Results May Appear Stale

If query results look outdated after restore, the cause is almost always client-side or agent-side caching, not a server-side index problem:

• Agent HTTP API response caching (default TTL varies by version)
• DNS resolver TTLs on the client or intermediate resolvers
• Stale blocking-query responses from a previous index
• A partially restored cluster where servers have not yet converged on the restored Raft log

Minimal Action to Confirm Fresh Results

No agent restart or manual rebuild is needed. To bypass caches and verify the server-side state:

1. Execute the prepared query with consistent reads and cache bypass:

   curl "http://localhost:8500/v1/query//execute?consistent=true&cached=false"

2. For DNS-based queries, use a fresh resolver or flush local DNS cache (e.g., systemd-resolve --flush-caches or dscacheutil -flushcache).
3. Compare results against the catalog and health endpoints for the same services:

   curl "http://localhost:8500/v1/catalog/service/"
   curl "http://localhost:8500/v1/health/service/"

Verification Checklist

1. Inspect the snapshot before restore: consul snapshot inspect <file> — confirm it contains the expected prepared query definitions.
2. After restore, confirm cluster convergence: Run consul operator raft list-peers and consul info on each server. Verify a single leader exists and all servers report matching raft.applied_index values.
3. Execute the prepared query with consistent reads (as above) and diff results against the expected catalog state from the snapshot.
4. Rule out client-side staleness by bypassing all caches as described.

Edge Cases Requiring Attention

• WAN federation: Prepared queries with failover to remote datacenters may return unexpected results if WAN state changed between snapshot time and restore time.
• Older agent versions: Agent HTTP API caching behavior is version-sensitive; agents on older versions may serve cached results briefly after restore.
• Partial restore: Restoring into a running cluster without ensuring all servers converge on the restored Raft log is the most common cause of inconsistent query answers.

Summary

There is no prepared-query index to rebuild. The restored Raft state is the source of truth. If results look wrong, the cluster has not fully converged or a cache layer is serving stale data. Bypass caches, verify Raft convergence, and re-run the query with consistent=true&cached=false to confirm correctness.