Question
Intermittent pg pool exhaustion: leak, event-loop stall, or misconfiguration?
Ari Orbit
0 reputation · 04 Jan 2023, 13:08 UTC
51K views1
Loading…
0 reputation · 04 Jan 2023, 13:08 UTC
Diagnosing intermittent connection pool exhaustion in node-postgres (pg) is complicated by the fact that multiple root causes produce the same 'timeout exceeded when trying to connect' error. A leaked client (missing release) and an event-loop stall that delays release callbacks can both leave the pool empty, as can a pool size that is too small for burst concurrency. The symptoms are further masked when application-level retries hide the underlying pattern.
A documented but often overlooked behavior is that pg does not automatically reap leaked clients; they remain checked out until the pool is destroyed. This leads to an unresolved decision: should the pool enforce a hard acquire timeout (connectionTimeoutMillis) to fail fast, or should the application rely on external retry logic to absorb transient backend issues? The trade-off is between surfacing failures quickly and hiding temporary conditions that might resolve on their own.
Which diagnostic signal most reliably distinguishes a leak from an event-loop stall? And under what conditions does a pool-level timeout become more harmful than helpful?
35775 reputation · 04 Jan 2023, 18:04 UTC
The diagnostic signal that most cleanly separates a connection‑leak from an event‑loop stall is the relationship between pool checkout/checkin counts and request latency.
If the pool is exhausted because connections are never returned, the checkout count will rise steadily while the checkin count lags or stays flat, producing a monotonic increase in "checked-out" connections even when request latency is normal.
nodejsacquire and release counters.pg_stat_activity for the application user: many idle in transaction connections point to unfinished transactions (stall‑related), whereas many idle connections that never disappear suggest a leak.Do you observe concurrent request latency spikes when the pool exhaustion occurs? Knowing this will tell us whether to focus on async‑task scheduling (stall) or on missing client.release() calls (leak).
Setting a low connectionTimeoutMillis turns temporary stalls or brief backend latency into hard failures, causing unnecessary retries and masking the chance for the stall to resolve. It is harmful when:
In those scenarios it is better to rely on the application’s retry logic or to increase the timeout while fixing the root cause (stall or leak).
Use comments to ask for clarification. Post a solution as an answer.
35,775 reputation · 05 Jan 2023, 01:05 UTC
Leaked clients cause a steady rise in the number of checked‑out connections that never drops, visible as a monotonic trend in the pool’s used metric. An event‑loop stall, by contrast, produces periodic spikes that line up with long‑running queries; the pool usage returns to baseline once the stall ends. Recording a histogram of checkout durations (or using pg_stat_activity to see how long each backend has been idle in transaction) makes the two patterns distinguishable.