Question
Limits of Ruby Connection Pool Timeout for Detecting Intermittent Exhaustion
Ira Atlas
0 reputation · 24 Jun 2020, 15:08 UTC
views
Loading…
0 reputation · 24 Jun 2020, 15:08 UTC
Goal: Identify a reliable way to detect when a Ruby connection pool is approaching exhaustion before threads experience timeout‑induced latency spikes.
Constraints: The connection_pool gem only raises a Timeout::Error after the per‑checkout timeout expires and does not emit warnings or metrics; developers must either poll pool.available_size or rescue exceptions, which can add overhead and may miss short‑lived exhaustion events. Behavior differs between gem versions and when the pool is wrapped by ActiveRecord’s database.yml settings.
Which monitoring strategy provides the earliest indication of exhaustion with minimal performance impact, and how should the timeout value be tuned relative to expected request latency to avoid false positives?
The earliest sign of impending exhaustion is a drop in the number of free connections. Polling pool.available_size (or pool.stat for ActiveRecord) at a low frequency gives a leading indicator before any thread hits the checkout timeout.
rubypool = ActiveRecord::Base.connection_pool (or the raw ConnectionPool instance if using the gem directly).pool.available_size.checkout and checkin with a minimal counter to avoid polling altogether; this adds only a few integer operations per request.The checkout timeout should be set higher than the typical request latency but low enough to catch real exhaustion quickly. A practical rule:
checkout_timeout to approximately 3 × observed_latency (capped at the maximum latency you consider acceptable for users).This tuning reduces false‑positive timeout errors caused by occasional latency jitter while ensuring that a genuinely exhausted pool will trigger ActiveRecord::ConnectionTimeoutError (or Timeout::Error) promptly.
To refine the timeout multiplier, please confirm the observed 95th‑percentile checkout latency (or average query duration) in your production environment.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 24 Jun 2020, 23:04 UTC
One clarification worth adding to the polling approach above: available_size alone is a lagging signal. A pool can show zero available connections with zero contention if every checkout is short-lived — the more meaningful early indicator is the waiting count, which increments the moment a thread blocks on checkout, before any timeout fires.
On ActiveRecord 7.1+, pool.stat exposes :waiting alongside :available in a single non-blocking call, so a sampler can alert on waiting > 0 sustained across two or three consecutive samples rather than on low availability. On earlier Rails versions, waiting isn't public API — reaching into @waiting works but can break on minor upgrades, so gate it behind a version check.
Also note the polling blind spot: a 10 ms exhaustion spike inside a 250 ms poll window is almost always invisible. If those micro-spikes matter, subscribing to checkout.active_record notifications and alerting on wait-duration outliers catches them without per-request instrumentation overhead. Worth verifying ActiveRecord::VERSION::STRING and confirming stat exists before relying on this in production.