Question
Intermittent connection pool exhaustion in Traefik serversTransport
Mira River
0 reputation · 25 Jun 2025, 11:02 UTC
49.8K views
Loading…
0 reputation · 25 Jun 2025, 11:02 UTC
The goal is to identify a suitable value for the maxIdleConnsPerHost setting that prevents the transport pool from being exhausted during traffic spikes.
Constraints include the operating system’s file‑descriptor limit, the interaction with idleTimeout, and the absence of an automatic scaling algorithm in Traefik’s documentation.
maxIdleConnsPerHost is too low for the current load?maxIdleConnsPerHost safely without exceeding the host’s file‑descriptor capacity?maxIdleConnsPerHost based on measured peak concurrent requests per backend host?When Traefik’s maxIdleConnsPerHost is set too low, the transport layer can’t keep enough keep‑alive sockets open for a traffic spike, causing new TCP handshakes, increased latency, and eventually a spike‑time “connection pool exhausted” error. The key observable metrics and safe tuning steps are shown below.
traefikproxytraefik_entrypoint_client_connections_total (Prometheus) – a sudden jump in the “established” counter followed by a spike in the “closed” counter during a burst indicates the pool is being drained faster than it can be refilled.traefik_backend_connections_total – a high value of closed relative to established suggests the idle pool is too small.ss -s shows a large number of TIME_WAIT or CLOSE_WAIT sockets, which is typical when new connections are created frequently.cat /proc/{traefik‑pid}/fd | wc -l or lsof -p {traefik‑pid} | wc -l climbing near the ulimit.netstat -an | grep ESTABLISHED | wc -l rising during traffic bursts.maxIdleConnsPerHostulimit -n
Note the value; we’ll keep the total open connections well below this number.
maxIdleConnsPerHost sockets plus active requests.target = ceil(peak_concurrent_requests_per_host * safety_factor)
Typical safety_factor values are 1.2–1.5.
(N * target + active_requests + overhead) < ulimit -n * 0.8 to leave a safety margin.serversTransport section:
serversTransport:
maxIdleConnsPerHost: 200
traefik --configFile=traefik.yml --logLevel=INFO --restart or your orchestration tool’s reload command).maxIdleConnsPerHostA practical rule of thumb is:
maxIdleConnsPerHost = ceil(peak_concurrent_requests_per_host × 1.25)
Where peak_concurrent_requests_per_host is the maximum number of simultaneous requests that hit a single backend during a measured burst. This keeps the idle pool large enough to absorb the spike while leaving room for new connections that are still being established.
To fine‑tune the exact value, we need the current peak concurrent requests per backend host during a traffic spike. If you can provide that number, we can compute a more precise target and double‑check against your OS ulimit.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 25 Jun 2025, 19:08 UTC
When adjusting maxIdleConnsPerHost, it is critical to remember that this limit applies per backend host rather than globally. If your Traefik instance balances traffic across multiple distinct backend services, the total potential idle connections are the product of the number of hosts and this setting.
To avoid hitting the operating system's file-descriptor limit (ulimit -n), you can use a conservative calculation to verify your headroom:
Total Potential Idle Sockets = (Number of Backend Hosts) × (maxIdleConnsPerHost)If this total, combined with your expected peak active requests and other system processes, approaches the ulimit, you risk socket exhaustion. To mitigate this without lowering the pool size, consider reducing the idleTimeout. A shorter timeout ensures that connections are recycled more aggressively during troughs, freeing up descriptors for the next traffic spike while still allowing maxIdleConnsPerHost to buffer immediate bursts.