ActiveRecord connection pool exhaustion intermittently under multithreaded requests

Root Cause: Pool Sizing vs. Process Model

ActiveRecord creates a separate connection pool per process. With the default pool: 5 and a multi-process server (Puma clustered, Passenger), each worker gets its own pool of five connections. If max_threads exceeds five, or if async queries (Rails 7+) hold connections during I/O wait, threads block on checkout and eventually raise ActiveRecord::ConnectionTimeoutError after the default five-second timeout. The logs show the timeout exception but no explicit "pool empty" message, making saturation look like latency.

Rails Version-Specific Thread-Safety Changes

• Rails 6.0: Rewrote the pool (ConnectionPool::Queue) with strict FIFO checkout and removed the legacy @waiting list.
• Rails 6.0–6.1: A bug in reaping_frequency (default 60s) delayed dead-connection cleanup, inflating busy counts. Fixed in 6.1.4/7.0.1; set reaping_frequency: 10 in database.yml if on affected versions.
• Rails 7.0+: Adds load_async which holds connections until the future resolves, increasing effective contention.
• Rails 7.1+: Introduces config.active_record.async_query_executor (default :global_thread_pool) — those threads also check out connections, further raising demand.

Distinguishing Saturation from Latency

Enable debug logging temporarily (config.log_level = :debug) to see Checkout/Checkin events with thread IDs and timestamps. Then inspect the pool state via ActiveRecord::Base.connection_pool.stat which returns { size:, connections:, busy:, dead:, waiting:, checkout_timeout: }.

• True exhaustion: busy == size and waiting > 0.
• Latency-driven timeouts: busy < size but timeouts occur — indicates slow queries, external API calls, GC pauses, or network stalls holding connections longer than checkout_timeout.

Reliable Metrics for Intermittent Exhaustion

1. pool.stat[:waiting] sampled periodically (via Prometheus exporter or middleware) — sustained > 0 means undersized pool or leaks.
2. Rate of ConnectionTimeoutError.
3. Average checkout duration (instrument connection.active_record notifications).
4. DB-side active connections (pg_stat_activity or SHOW PROCESSLIST) vs. pool × processes.

Verification Steps

# Health endpoint (add to routes)
get '/pool-stat', to: ->(env) { [200, { 'Content-Type' => 'application/json' }, [ActiveRecord::Base.connection_pool.stat.to_json]] }

# Debug logging (staging only)
config.log_level = :debug

# Checkout duration instrumentation
ActiveSupport::Notifications.subscribe('connection.active_record') do |*args|
  event = ActiveSupport::Notifications::Event.new(*args)
  if event.payload[:name] == 'checkout'
    Rails.logger.info "checkout_duration=#{event.duration}ms"
  end
end

# DB-side check (PostgreSQL)
SELECT count(*) FROM pg_stat_activity WHERE state = 'active';

# Load test
wrk -t4 -c100 -d30s http://staging.example.com/pool-stat

One Missing Diagnostic Detail

What is the exact Rails version and config.active_record.async_query_executor setting? Rails 7.1+ with the global thread pool adds hidden connection demand that changes the pool-sizing calculation.