Question
Application Connection Pool Interaction with Linux TCP TIME_WAIT Reuse Under High Connection Churn
Ari Dey
0 reputation · 25 Sept 2020, 14:37 UTC
112.1K views1
Loading…
0 reputation · 25 Sept 2020, 14:37 UTC
Goal: Determine whether enabling the kernel option net.ipv4.tcp_tw_reuse alleviates intermittent connection‑pool exhaustion observed in high‑churn TCP workloads when the application pool evicts idle connections based on a timeout.
Constraints: The application’s connection pool maintains a maximum size and evicts idle sockets after a configurable idle‑timeout, leaving sockets in the ESTABLISHED state until the pool limit is reached. Enabling tcp_tw_reuse may allow the kernel to reuse TIME_WAIT sockets for new outbound connections, but it is only safe for protocols that tolerate relaxed TCP ordering and may affect reliability if delayed packets are accepted.
Uncertainty: It is unclear how the interaction between the pool’s idle‑timeout eviction and the kernel’s TIME_WAIT reuse influences the steady‑state counts of ESTABLISHED and TIME_WAIT sockets, and whether the observed exhaustion symptoms are primarily driven by port‑allocation limits or file‑descriptor consumption.
Does enabling net.ipv4.tcp_tw_reuse reduce the frequency of bind() errors due to ephemeral‑port exhaustion without causing a corresponding rise in ESTABLISHED sockets held by the pool?
Given a fixed pool max size and idle‑timeout, what is the threshold of connection rate at which TIME_WAIT reuse begins to offset pool‑exhaustion symptoms, and how does this threshold shift when tcp_fin_timeout is tuned?
35275 reputation · 25 Sept 2020, 19:05 UTC
Enabling net.ipv4.tcp_tw_reuse=1 reduces bind() failures from ephemeral-port exhaustion without increasing the pool's ESTABLISHED socket count, provided the workload is outbound-client dominated and TCP timestamps are active on both endpoints. The mechanism reuses a TIME_WAIT socket for a new outbound connection instead of allocating a fresh port, so the concurrent ESTABLISHED count stays bounded by the pool's configured maximum.
linuxtcp_tw_reuse allows the kernel to recycle a socket in TIME_WAIT for a new SYN when the 4-tuple (src/dst IP/port) is safe to reuse. This replaces a closing socket; it does not create an extra concurrent ESTABLISHED connection.net.ipv4.tcp_timestamps=1 (default on modern kernels) and that middleboxes do not strip the TCP timestamp option. Without timestamps, the kernel cannot verify PAWS (Protection Against Wrapped Sequence numbers) and will not reuse.net.ipv4.tcp_fin_timeout (default 60 s) shortens the FIN-WAIT-2 state, reducing the inflow into TIME_WAIT. This shifts the exhaustion threshold to higher connection rates.Your pool evicts idle connections after a timeout, sending FIN and entering FIN-WAIT-1 → FIN-WAIT-2 → TIME_WAIT. Under high churn, the rate of new outbound connections exceeds the rate at which TIME_WAIT sockets expire (default 60 s, controlled by tcp_fin_timeout for the preceding state and a fixed 60 s TIME_WAIT per RFC 793). The ephemeral port range (net.ipv4.ip_local_port_range, typically 32768–60999 ≈ 28k ports) becomes saturated, causing EADDRNOTAVAIL on bind(). The pool's ESTABLISHED count remains at its configured max because evictions are balanced by new acquisitions—exhaustion is a port-availability problem, not a file-descriptor or pool-size problem.
The steady-state TIME_WAIT count ≈ connection_rate × TIME_WAIT_duration. With default 60 s TIME_WAIT and ~28k ports, the theoretical ceiling is ~460 connections/sec before ports recycle naturally. Enabling tcp_tw_reuse effectively reduces the TIME_WAIT residency to the RTO (retransmission timeout, ~200 ms–3 s), raising the ceiling to ~10k–140k connections/sec. Tuning tcp_fin_timeout down (e.g., to 15 s) cuts the FIN-WAIT-2 contribution, but the dominant factor remains the TIME_WAIT duration, which tcp_tw_reuse bypasses.
sysctl net.ipv4.tcp_timestamps (must be 1).sysctl net.ipv4.ip_local_port_range.ss -s or netstat -ant | awk '/TIME_WAIT|ESTABLISHED/ {print $1}' | sort | uniq -c.Cannot assign requested address or EADDRNOTAVAIL during peaks.tcp_tw_reuse temporarily: sysctl -w net.ipv4.tcp_tw_reuse=1 and re-measure.What is the observed peak outbound connection rate (connections/sec) during exhaustion events? This determines whether the port range is the bottleneck or if file-descriptor limits (ulimit -n, fs.file-max) are being hit first.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 25 Sept 2020, 16:54 UTC
One clarification worth adding before tuning anything: net.ipv4.tcp_fin_timeout controls how long a socket lingers in FIN_WAIT_2, not the TIME_WAIT duration itself. TIME_WAIT length is nominally fixed at 2×MSL (60 s) in the kernel and isn't directly tunable via that sysctl. So if your ss -tan output shows TIME_WAIT dominating during exhaustion events, adjusting tcp_fin_timeout won't move the threshold much — it only helps connections stuck in FIN_WAIT_2 (typically when the peer never sends its FIN).
Also worth separating the two failure modes empirically: correlate the application's exact errors. EADDRNOTAVAIL on connect points to ephemeral-port exhaustion (what tcp_tw_reuse addresses); EMFILE/ENFILE points to file-descriptor limits, which it won't touch. A quick ss -tan state time-wait | wc -l versus state established under load, plus checking ip_local_port_range size against your churn rate × 60 s, will tell you which ceiling you're actually hitting.