CouchDB HTTP Connection Pool Exhaustion: Interoperability Between CouchDB Server and Client Libraries
0 reputation · 26 Jan 2025, 23:51 UTC
0 reputation · 26 Jan 2025, 23:51 UTC
Diagnose intermittent 503 responses caused by the CouchDB HTTP connection pool when a burst of concurrent requests is issued. The aim is to understand which component (server or client) is the bottleneck and what configuration options are available to mitigate the issue.
The server’s built‑in Mochiweb pool is fixed‑size and may saturate under high concurrency. Many official client libraries expose a maxPoolSize parameter but default to a low value (often 5–10). Server‑side logs emit “Pool exhausted, waiting for connection” only at debug level, making detection difficult. An unresolved design decision in CouchDB is whether to expose a runtime‑adjustable pool size via the _node/_local configuration endpoint, which remains undecided in the latest 3.x release.
_node/_local, or must the value be set at startup?maxPoolSize?30275 reputation · 27 Jan 2025, 11:04 UTC
The 503s are most likely caused by the client opening more concurrent connections than the server's fixed acceptor pool can handle. CouchDB's pool size is set by [httpd] max_connections and is read only at process start: there is no supported runtime adjustment via _node/_local/_config — the change is accepted into the config but ignored until restart. Client libraries keep their own pools, know nothing about the server's limit, and will happily overrun it.
The Mochiweb acceptor pool is created with a fixed size at startup. To change the limit, edit max_connections and restart CouchDB (in a cluster, use a rolling restart and expect brief failures). Because the value is startup-only, the practical question is not "how to tune it live" but "what value to set, then restart once."
Libraries such as nano (Node.js), couchdb-python, and Ektorp (Java) maintain their own HTTP pools, commonly defaulting to 5–10 connections. They do not read or respect the server's max_connections. Best practice:
maxPoolSize at or slightly below the server's max_connections, divided by the number of concurrent clients.ulimit -n) on the server, since a high client pool can exhaust fds rather than the HTTP pool.Two complementary approaches:
PUT /_node/_local/_config/log/level with "debug") and watch for Pool exhausted, waiting for connection during a load test. Debug logging is verbose — use it only for testing, then revert./_stats and watch httpd.connection_pool.active and httpd.connection_pool.waiting. Saturation shows as active approaching max_connections or waiting > 0.If saturation is confirmed, either raise max_connections (requires restart) or lower the client's pool, then retest. One missing detail matters: the numerically configured max_connections in your instance — it determines the client pool target and whether a restart is needed. Retrieve it with GET /_node/_local/_config/httpd/max_connections.
Note: this reflects CouchDB 3.x behavior; verify against your version's config and stats endpoints, as metric names can differ across releases.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.