Consul LAN gossip port limits for multi-agent clusters

Answer first

TCP-only reachability on 8301 does not satisfy stable LAN membership. Serf LAN gossip uses port 8301 for both UDP and TCP, with UDP as the primary transport for gossip dissemination and failure detection. TCP is also used by Serf.

With gossip encryption, an agent accepts messages encrypted with any key listed in the encrypt configuration, but transmits messages encrypted with the first key in the list. During rotation, transmit key selection is strictly the first key for that agent.

Confirmed facts

• Consul Serf LAN gossip is port 8301 for both UDP and TCP. UDP is primary for gossip and failure detection; TCP is also used.
• TCP-only reachability on 8301 is insufficient for reliable LAN membership.
• Encrypt list behavior is agent-local: accept any listed key, transmit with the first key in the list.

Likely explanation for partial reachability

Partial reachability, e.g., UDP 8301 blocked but TCP 8301 allowed between a subset of agents, leads to degraded gossip convergence, increased suspicion timeouts, inconsistent member lists and possible partitions. Full bidirectional UDP and TCP 8301 between agents is needed for reliable membership. Symptoms can mimic latency or resource issues and are not visible from server RPC 8300 health alone.

Steps needed for this case

1. Verify bidirectional reachability for both UDP and TCP on port 8301 for all agent pairs, not just TCP 8300 server RPC.
2. Run consul members on each agent and compare outputs. Persistent divergence indicates a gossip partition.
3. Inspect running agent configuration for the encrypt list order and confirm the first key matches the intended transmit key during rotation. Ensure ordering is coordinated across the fleet so all agents can receive with old keys and transmit with the intended current key.
4. Monitor membership views for flapping or divergence during the rolling change.

One missing diagnostic detail that changes the recommendation: the Consul version in use and the current encrypt list ordering across the fleet. Older releases may differ in TCP fallback details, and key ordering is agent-local, so rotation safety depends on consistent ordering.