nginx reverse proxy ↔ upstream application servers: which side's idle timeout governs safe connection reuse?

Direct Answer

The upstream application server's idle timeout must be strictly greater than nginx's keepalive_timeout (in the upstream block). If the upstream timeout is less than or equal to nginx's, a deterministic race window exists where nginx hands out a connection the upstream has already closed, causing sporadic 502/504 errors that mimic pool exhaustion.

Deterministic Guard Combination

Configure nginx's upstream-block keepalive_timeout at least 5–10 seconds below the upstream server's configured idle timeout. For example, if your upstream (Tomcat, Gunicorn, Node, Go http.Server, etc.) is set to 60 seconds idle:

upstream backend {
    server 10.0.0.1:8080;
    keepalive 100;                 # per worker
    keepalive_timeout 50s;         # evict from pool BEFORE upstream closes
    keepalive_requests 100;        # default, but set explicitly
    keepalive_time 50s;            # nginx ≥1.15.3: time-based eviction
    # proxy_http_version 1.1;      # required in location block
    # proxy_set_header Connection "";
}

This ensures nginx evicts idle connections from its pool before the upstream sends FIN. The race is eliminated because nginx never sees a pooled connection that the upstream has already closed.

Why This Works

nginx does not probe or validate pooled connections before reuse (no TCP keepalive by default, no application-level ping). When a request arrives, nginx pulls a connection from the pool and writes immediately. If the upstream closed that connection during its idle period, the write fails with ECONNRESET/ETIMEDOUT, logged as a 502/504 with non-zero $upstream_connect_time.

The keepalive_requests cap and keepalive_time (nginx 1.15.3+) provide secondary guards. Set keepalive_requests below any upstream max-requests-per-connection limit. Set keepalive_time to match or be less than keepalive_timeout to force periodic recycling.

Measuring Pool Reuse Rate in Staging

Parse the access log for $upstream_connect_time. A value of 0.000 (or near-zero) indicates connection reuse; a positive value indicates a new connection was established.

# Reuse rate = zero-connect / total
awk '{print $NF}' /var/log/nginx/access.log | \
  awk '$1 == "0.000" {zero++} {total++} END {print zero/total*100 "%"}'

For richer metrics, enable a status zone (nginx Plus or third-party module) and query /status for keepalive counters. Under sustained load (> upstream idle timeout), watch for 502/504 spikes at multiples of the upstream timeout — that pattern confirms the race.

One Diagnostic Detail Needed

What is the exact idle timeout configured on your upstream application server? (Not the default — the value in its config: Tomcat keepAliveTimeout, Jetty IdleTimeout, Gunicorn --keep-alive, Node server.keepAliveTimeout, Go IdleTimeout, etc.) This single number determines the safe nginx keepalive_timeout ceiling. If an intermediate load balancer, API gateway, or TLS terminator sits between nginx and upstream, its idle timeout becomes the effective limit instead.