Django REST Framework and PostgreSQL: Managing Connection Pool Exhaustion with CONN_MAX_AGE
0 reputation · 25 Mar 2023, 12:21 UTC
0 reputation · 25 Mar 2023, 12:21 UTC
In high-traffic Django REST Framework (DRF) deployments using PostgreSQL, the CONN_MAX_AGE setting determines whether database connections are closed at the end of a request or persisted for future use. While persistent connections reduce the overhead of repeated handshakes, they can lead to connection exhaustion if the number of concurrent worker processes exceeds the database server's max_connections limit.
Django does not natively manage a driver-level connection pool (such as psycopg2's internal pooling mechanisms) through standard settings. This creates a gap between the application's request-handling lifecycle and the database's available connection slots, especially when long-running API views hold connections open.
Given these constraints, how should the balance between CONN_MAX_AGE and external pooling tools like PgBouncer be configured to prevent intermittent exhaustion? Does the current architecture allow for a configurable pool size within the Django settings, or is external middleware the only viable solution for limiting total connections?
32775 reputation · 25 Mar 2023, 14:22 UTC
Django does not have a configurable internal connection pool size; it only supports connection persistence per process. To prevent exhaustion, you must use an external pooler like PgBouncer if your total worker count (processes × threads) exceeds PostgreSQL's max_connections. When using PgBouncer in transaction mode, CONN_MAX_AGE should be set to 0 (the default) to ensure connections are released back to the pool immediately after the request.
It is critical to distinguish between CONN_MAX_AGE and true connection pooling. CONN_MAX_AGE enables persistence: it tells a specific Django worker process to keep its single database connection open for a set duration. It does not allow workers to share a pool of connections.
In a high-traffic DRF environment using a multi-process server (like Gunicorn), the math is linear:
Total Connections = (Number of Worker Processes) × (Threads per Worker)
If you set CONN_MAX_AGE to a high value, every worker holds a slot in PostgreSQL even when idle. If your worker count exceeds the database max_connections limit, you will encounter too many clients already errors.
Use this only for low-to-medium scale deployments where you can guarantee the worker count is well below max_connections.
CONN_MAX_AGE to a low value (e.g., 60 seconds) to reduce handshake overhead without holding slots indefinitely.max_connections in postgresql.conf is higher than your total possible worker threads.This is the standard for high-traffic DRF apps. PgBouncer sits between Django and PostgreSQL, multiplexing many application connections into a few database connections.
transaction mode for maximum efficiency.CONN_MAX_AGE = 0. Since PgBouncer manages the pool, Django should close its connection to the proxy at the end of the request to allow the proxy to reassign that slot.To verify current connection usage, run the following query on your PostgreSQL instance:
SELECT count(*), state FROM pg_stat_activity GROUP BY state;
If the number of idle connections matches your total worker count, CONN_MAX_AGE is effectively pinning those slots.
Missing Diagnostic: Are you using Gunicorn/uWSGI with a threaded or eventlet/gevent worker class? This significantly changes the number of concurrent connections generated per process.
Use comments to ask for clarification. Post a solution as an answer.
32,775 reputation · 25 Mar 2023, 17:01 UTC
Starting with Django 5.1, when the database backend uses psycopg (≥3), the DATABASES OPTIONS dict can include a pool key (e.g., {'min_size': 2, 'max_size': 10, 'timeout': 10}). This creates a per‑worker connection pool that is independent of CONN_MAX_AGE. When a native pool is active, setting CONN_MAX_AGE to a positive value is unnecessary and can interfere with the pool’s lifecycle; the recommended practice is to keep it at the default 0 so each checkout returns to the pool promptly after the request.
You can verify the pool is active by checking django.db.backends.base.BaseDatabaseWrapper’s pool attribute after the first connection.