Fixed versus adaptive suspicion multiplier in Consul's WAN gossip failure detector

The short answer

Neither approach wins universally. A fixed suspicion multiplier gives you predictable, steady detection latency; an adaptive multiplier gives you fewer false positives on volatile WAN links. The factors that most strongly decide which one performs better are cluster size, WAN round-trip time variance, and how well the gossip interval is already tuned. The multiplier cannot compensate for a badly set gossip interval — tune that first.

What the multiplier actually does

Consul's failure detector (built on SWIM-style gossip via Serf) marks a node suspect after a timeout derived from the gossip interval, scaled by the suspicion multiplier. Only after further missed probes does the node get declared failed. So the multiplier is a direct dial on the speed/accuracy trade-off:

• Lower multiplier → faster failure detection, more false positives when a single probe is delayed.
• Higher multiplier → slower detection, more tolerance for transient latency spikes.

Fixed multiplier: strengths and failure modes

A fixed value keeps detection latency constant and easy to reason about — useful for capacity planning and alerting SLAs. Its weakness is that WAN gossip delays are not constant. When cross-datacenter round-trip times spike (congestion, intermittent partitions), a static threshold no longer matches real connectivity, and healthy nodes start getting suspected. In large clusters the problem compounds: gossip propagation latency grows with membership, so a fixed multiplier tuned on a small cluster tends to fire prematurely at scale.

Adaptive multiplier: strengths and failure modes

An adaptive strategy scales suspicion using recent gossip round-trip statistics, so transient latency spikes raise the effective threshold instead of producing false positives. The risks are on the other side:

• If damping or smoothing is too aggressive, genuine failures take longer to confirm — detection latency suffers exactly when you need it.
• Without enough sampling depth, adaptation overfits to short-term noise and oscillates.
• Behavior is harder to predict and audit than a fixed number, which matters for incident postmortems.

Where the performance boundary sits

As a rule of thumb:

• Stable WAN links, small-to-medium clusters, strict detection-time SLAs → fixed multiplier is simpler and adequate.
• Volatile links, large membership, or frequent transient partitions → adaptive earns its complexity by cutting false-positive churn (flapping suspect/failed events that trigger needless rerouting and alerts).

Note that this is the likely explanation based on how SWIM-family detectors behave generally. Confirmed facts are limited: Consul exposes the multiplier via agent configuration, and exact suspicion algorithm details can change between releases — validate against your specific agent version before committing.

Verifying in your own cluster

1. Baseline the WAN: run consul members -wan and measure RTTs between datacenters under normal and stressed load.
2. Correlate agent logs: timestamp suspect events against your configured gossip interval to see whether the multiplier or the interval is the real problem.
3. A/B test in staging: run fixed vs adaptive settings against the same failure scenarios (kill a node, inject latency) and record detection latency and false-positive counts per configuration.

One diagnostic detail changes the recommendation: your measured WAN RTT variance, not the mean. If p99 RTT is close to the mean, stay fixed; if p99 is several times the mean, adaptive is the safer default.