Nomad client connection pool stale-connection eviction during leadership transitions

Direct answers

1. The client does not proactively close gRPC connections to a former Raft leader; those connections remain open until idle_timeout expires.
2. max_wait is strictly a pool-slot acquisition budget. It does not cover leader re-discovery, Consul/DNS resolution, or TCP/TLS dialing to the new leader.
3. No universal idle_timeout floor is documented for frequent leadership changes. A shorter value can release stale slots sooner, but setting it too low may churn healthy connections. Confirm your Nomad version before changing it.

Confirmed facts versus assumptions

Item	Status	Notes
Proactive close on leadership loss	Reported, not version-verified	Client relies on idle_timeout rather than leader-change events.
max_wait scope	Reported, not version-verified	Covers slot wait only; leader discovery and dialing are outside it.
Stale connections during flaps	Likely explanation	Former-leader connections consume max slots until idle_timeout.
Per-target connection metrics	Confirmed gap	No built-in metric exposes per-target counts or eviction events.

Steps for this case

1. Record the Nomad client and server versions. Eviction behavior may differ across versions, so this determines whether the guidance applies.
2. Review the client connection_pool stanza:

client {
  connection_pool {
    min = 2
    max = 64
    idle_timeout = "30s"
    max_wait = "5s"
  }
}

1. If leadership flaps are frequent, consider lowering idle_timeout modestly, for example toward 10–15 seconds, so stale former-leader connections release sooner. Test in staging first and avoid values below the typical election plus dial time.
2. Do not use max_wait to absorb leadership transition latency. If callers time out during flaps, increase caller timeouts or reduce flap frequency instead of only tuning max_wait.
3. Verify after a leadership change:

• Inspect Nomad client logs for “connection evicted” or “connection closed” entries referencing the old leader address.
• Compare min and max against active connections shown in the Nomad UI or API for the client node.

Missing diagnostic detail

What Nomad version are the clients and servers running? This detail changes whether the recommended idle_timeout adjustment is safe for your cluster.

Verification

• Check client logs after a known leadership change for “connection evicted” or “connection closed” entries tied to the former leader.
• Use the Nomad UI or API to list active client connections and confirm whether former-leader connections remain open past idle_timeout.

This is a conservative draft for human review because the underlying behavior requires verification against your specific Nomad version.