"FATAL: sorry, too many clients already" from NetBox under load: Gunicorn workers vs PostgreSQL max_connections

Likely explanation

Each Gunicorn sync worker opens a new PostgreSQL connection for every request when DATABASE_CONN_MAX_AGE is 0 (the NetBox default). Under load, the total number of concurrent connections approximates:

workers × average simultaneous requests per worker

When this product exceeds the PostgreSQL max_connections setting, the server replies with FATAL: sorry, too many clients already, which Django surfaces as an OperationalError.

Confirmed facts

• Raising max_connections or lowering the Gunicorn worker count directly reduces the chance of hitting the limit.
• Setting CONN_MAX_AGE > 0 enables connection reuse inside a worker, lowering new connections per request, but can cause server closed the connection unexpectedly if PostgreSQL restarts or the idle timeout exceeds the age.
• Async Gunicorn workers (e.g., gevent, eventlet) can share a single connection pool across many concurrent requests, lowering the connection count, but NetBox plugin compatibility is not guaranteed and should be tested.

Steps to resolve for this deployment

1. Measure current usage: During a representative load spike, run

   SELECT count(*) FROM pg_stat_activity;

   and compare the result to SHOW max_connections; to confirm exhaustion.
2. Determine concurrency per worker: Obtain the average number of simultaneous requests each Gunicorn worker handles (e.g., via Gunicorn stats, Prometheus, or gunicorn --statsd-host). This figure is needed to decide whether adjusting workers or max_connections is more appropriate.
3. Choose a mitigation:
   • If you have headroom on the database server, increase max_connections to safely accommodate workers × concurrency.
   • If raising max_connections is undesirable (memory/context‑switch concerns), reduce the Gunicorn worker count so that the product stays below the limit.
4. Consider connection reuse: In a staging environment, set DATABASE_CONN_MAX_AGE=60 (or another suitable value) and verify that connection churn drops. Then simulate a PostgreSQL restart to ensure no server closed the connection unexpectedly errors appear.
5. Evaluate async workers (optional): If plugin compatibility is confirmed, test a small canary with gunicorn -k gevent (or eventlet) and monitor connection counts. Roll back if any plugin exhibits errors.

Missing diagnostic detail: To finalize the recommendation between lowering workers versus raising max_connections, please provide the measured average simultaneous requests per Gunicorn worker under load.