TCP Keepalive: Connection Pool Exhaustion and Stale Socket Recovery

Direct Answer

Use per-socket setsockopt calls with TCP_KEEPIDLE, TCP_KEEPINTVL, and TCP_KEEPPROBES to override the global keepalive timers for each connection in your pool. This avoids system-wide side effects and lets you tune aggressiveness per workload. The kernel defaults (tcp_keepalive_time=7200, tcp_keepalive_intvl=75, tcp_keepalive_probes=9) wait ~2 hours before probing and ~12 minutes to declare a peer dead—far too long for a high-density pool.

Recommended Per-Socket Values

For a typical long-lived idle session that must survive transient congestion:

• TCP_KEEPIDLE = 300 (5 minutes before first probe)
• TCP_KEEPINTVL = 30 (30 seconds between probes)
• TCP_KEEPPROBES = 5 (declare dead after ~2.5 minutes of no response)

This detects a crashed peer within ~3 minutes while tolerating brief network hiccups. Adjust KEEPIDLE upward (e.g., 600–900) if your legitimate sessions routinely idle longer than 5 minutes.

Applying Per-Socket (C Example)

int fd = socket(AF_INET, SOCK_STREAM, 0);
int optval = 1;
setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &optval, sizeof(optval));

optval = 300;  // seconds
setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &optval, sizeof(optval));
optval = 30;
setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &optval, sizeof(optval));
optval = 5;
setsockopt(fd, IPPROTO_TCP, TCP_KEEPPROBES, &optval, sizeof(optval));

These options are documented in man 7 tcp and the kernel source (Documentation/networking/socket.txt). They are stable since Linux 2.4; the TCP_KEEPIDLE/INTVL/PROBES names have been consistent since at least 4.14.

Language-Specific Notes

• Go: net.TCPConn.SetKeepAlivePeriod (since 1.15) sets idle+interval together; probes not directly exposed.
• Java: SocketOptions.TCP_KEEPIDLE etc. via setOption (JDK 11+).
• Python: socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 300) (constants available on Linux).
• Rust: socket2::TcpKeepalive builder.

If your runtime doesn't expose these, you must either accept the global sysctl or use a wrapper (e.g., LD_PRELOAD shim) to inject the options at connect time.

Global Fallback (If Per-Socket Impossible)

# /etc/sysctl.d/99-tcp-keepalive.conf
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5

Apply with sysctl --system. Warning: this affects all TCP sockets on the host, including SSH, database clients, and sidecar proxies. Aggressive values here can trigger stateful firewall timeouts or drop legitimate idle connections.

Verification Steps

1. cat /proc/sys/net/ipv4/tcp_keepalive_time (and _intvl, _probes) to confirm global values.
2. ss -tanp -o state established | grep -E 'timer:.*keepalive' shows per-socket keepalive timers (kernel 4.11+).
3. dmesg -T | grep -i keepalive after a test kill to see kernel timeout logs.

One Missing Detail

What language/runtime manages your connection pool? If it doesn't expose TCP_KEEPIDLE/INTVL/PROBES, the per-socket path may require a shim or force you to the global sysctl—changing the risk profile.