Consul Anti-Entropy Synchronization and Local State Divergence

Answer to the question

In a recent stable Consul release, the anti‑entropy mechanism is activated whenever a local agent records a health‑check state change (passing, warning, or critical). The agent immediately sends an RPC to the server cluster. That RPC triggers a gossip‑based gossip sync that propagates the new state to all peers, typically within about five seconds. The server is the authoritative source of truth for health state; when it receives a conflicting update from an agent, it compares the timestamps attached to each state record. The record with the most recent timestamp wins (last‑write‑wins). Updates older than the configurable stale_threshold (default 10 s) are ignored to prevent stale flapping data from re‑introducing old states.

During high‑frequency flapping, an agent may send many successive updates. The server coalesces these updates, keeping only the most recent timestamped record. The older updates are discarded, and the resolved state is pushed back to all agents via gossip.

How conflicts are resolved

• Server‑authoritative policy: The server always wins if its timestamp is newer.
• Last‑write‑wins: The server compares the check_id timestamps; the newer one is kept.
• Stale‑threshold filtering: Any update older than stale_threshold is ignored to avoid re‑introducing old states during rapid flapping.

Verification steps for this case

1. Check the agent log for state‑change and gossip entries:

   tail -f /var/log/consul/consul.log | grep -i "Health check state changed"
   

2. Confirm the server’s view of the service health:

   consul health checks -service <service_name>
   

3. Verify the server is the Raft leader (only the leader can commit state changes):

   consul operator raft list-peers | grep "Leader"
   

4. If the log shows repeated Health check state changed entries with timestamps older than stale_threshold, those updates are being dropped automatically.

One diagnostic detail that could change the recommendation

Is the server you are querying the current Raft leader? If not, the state changes you see may be from a follower that has not yet replicated the latest changes. In that case, you should query the leader or check the consul operator raft list-peers output for the Leader flag.