SQLAlchemy pool with DuckDB dialect: diagnosing intermittent exhaustion across forked workers
0 reputation · 02 Dec 2024, 04:34 UTC
0 reputation · 02 Dec 2024, 04:34 UTC
I am running DuckDB (assume v0.10.x) through the SQLAlchemy DuckDB dialect inside a multi-process web service (gunicorn-style workers). Connections are acquired from a SQLAlchemy QueuePool and released back after each request. Intermittently, the pool reports exhaustion even though the workload is well below the configured pool size.
DuckDB connections are in-process handles with no server-side limit, so exhaustion must come from the client pool or unreleased references. Two behaviors make this hard to pin down: the DuckDB Python module can keep a native connection alive until all derived cursors are released, and uncommitted transactions are silently rolled back on close(), which may hide application-level leaks. Additionally, each forked worker inherits its own DuckDB state, so a pool sized for the parent process may not match per-worker reality. I also understand the dialect may not enable pool_pre_ping by default, raising the question of whether stale connections after a fork are being counted as live.
close() affect when the native handle is actually released back for reuse, and is this consistent across v0.9 and v0.10?pool_pre_ping meaningful for an embedded engine like DuckDB, or is connection liveness purely a client-side reference-counting concern?A thoughtful contribution can make all the difference. Be the first to share one.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.