What telemetry can confirm whether intermittent Nomad client RPC failures stem from stale server connections?
0 reputation · 25 May 2020, 03:14 UTC
0 reputation · 25 May 2020, 03:14 UTC
Nomad clients maintain long-lived, multiplexed connections to Nomad servers for RPC traffic. In environments with NAT gateways, load balancers, or firewalls that silently drop idle TCP sessions, these connections can become stale without either side noticing immediately, which is one plausible cause of intermittent RPC latency or failures. The exact behavior is version-dependent and I have not verified it against a specific release.
The uncertainty is what a client actually does when a server connection goes stale: whether it detects the dead connection proactively (for example via application-level heartbeats or yamux keepalives) or only discovers it on the next RPC attempt, and whether it then reconnects automatically or surfaces repeated errors first.
I am also unsure which documented telemetry, if any, would let an operator distinguish "the pool of server connections is exhausted or stale" from unrelated causes such as server overload or network packet loss. Metric names and availability differ across Nomad versions, so any answer should state the version it applies to.
Does the Nomad client perform keepalive or health checking on its multiplexed server connections, and is any of that configurable? Which documented metrics or log entries would confirm stale-connection churn versus server-side slowness? Is there a supported way to bound connection lifetime or force periodic reconnection?
A thoughtful contribution can make all the difference. Be the first to share one.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.