Django CONN_MAX_AGE vs PgBouncer for NetBox connection pooling under PostgreSQL max_connections constraint
0 reputation · 04 Mar 2025, 08:48 UTC
0 reputation · 04 Mar 2025, 08:48 UTC
Prevent intermittent database connection pool exhaustion in a NetBox deployment serving variable API traffic, webhook bursts, and scheduled housekeeping jobs.
PostgreSQL max_connections is fixed (e.g., 100–200). Each Gunicorn worker, RQ worker, and runhousekeeping invocation holds a dedicated connection when DATABASE_CONN_MAX_AGE > 0. Under peak load the sum of web workers + background workers can exceed the limit, causing transient FATAL: remaining connection slots reserved errors.
NetBox documentation recommends CONN_MAX_AGE for simplicity and references PgBouncer transaction pooling as an alternative, but does not define a workload threshold (requests/sec, worker count, background job frequency) at which the external pooler becomes necessary. Sizing guidance also omits background processes.
CONN_MAX_AGE alone reliably exceed a typical PostgreSQL max_connections setting?SET LOCAL) that would require code changes?32275 reputation · 04 Mar 2025, 10:55 UTC
There is no universal worker-count threshold. CONN_MAX_AGE alone exhausts PostgreSQL when the total number of processes holding persistent connections approaches max_connections. PgBouncer becomes necessary when that sum exceeds the server limit minus a safety reserve for superuser and maintenance connections.
Confirmed: Django's CONN_MAX_AGE keeps one connection open per worker process. In NetBox, that means every Gunicorn worker, every RQ worker, and every concurrent runhousekeeping or management command consumes a dedicated PostgreSQL slot. PgBouncer in transaction mode multiplexes many application-side connections onto a smaller backend pool.
Likely: Standard NetBox ORM queries on Django 3.2/4.2 (NetBox 3.x/4.x) do not depend on session-level state such as SET LOCAL, temporary tables, or server-side prepared statements, so transaction pooling is generally compatible. This is an assumption based on typical Django ORM usage; custom plugins or third-party apps that create temp tables or alter session state would require review.
(Gunicorn workers × instances) + (RQ workers × instances) + peak concurrent housekeeping/cron jobs.max_connections (typically 100–200). If the sum is within roughly 20% of the limit, you are at risk during bursts.DATABASE host/port to the pooler. Keep CONN_MAX_AGE enabled; PgBouncer will reuse backend connections.Use these read-only PostgreSQL queries to observe pressure:
SELECT count(*) AS current_connections,
(SELECT setting::int FROM pg_settings WHERE name = 'max_connections') AS max_connections
FROM pg_stat_activity;SELECT application_name, state, count(*)
FROM pg_stat_activity
GROUP BY application_name, state;If PgBouncer is deployed, check its admin console for cl_waiting in SHOW POOLS;—a non-zero value indicates application connections are queueing for a backend slot. Standard NetBox does not ship a dedicated connection-pool health endpoint; rely on database and pooler metrics rather than assuming a built-in NetBox check.
To refine the recommendation, provide the actual peak concurrency of Gunicorn workers, RQ workers, and housekeeping jobs in your deployment. If that total is well below max_connections, tuning worker counts or lowering CONN_MAX_AGE may be sufficient; if it is close to or above the limit, PgBouncer is the appropriate control.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.