GitLab database pool configuration: missing cross‑component limit enforcement
0 reputation · 24 Oct 2022, 10:20 UTC
0 reputation · 24 Oct 2022, 10:20 UTC
Ensure that the combined database connection pools used by GitLab Rails/Puma, Sidekiq workers, and Workhorse never exceed the PostgreSQL server’s max_connections limit, preventing intermittent connection pool exhaustion under load.
gitlab_rails['db_pool'], sidekiq['db_pool'], and gitlab_workhorse['db_pool'].postgresql['max_connections'].What is the recommended approach to validate that the combined pool sizes remain within the PostgreSQL limit?
How can administrators automate detection of a potential overflow before applying configuration changes?
Should GitLab introduce a built‑in check or warning when the sum of the three pools exceeds the configured postgresql['max_connections']?
33275 reputation · 24 Oct 2022, 17:08 UTC
No — to the best of my knowledge, Omnibus GitLab performs no cross-component validation between the per-component pool settings and postgresql['max_connections']. gitlab-ctl reconfigure will apply a configuration whose combined pools exceed the server limit without complaint; the failure only surfaces at runtime, when PostgreSQL starts refusing connections. Validation is therefore a manual (or self-scripted) arithmetic check plus runtime monitoring. Treat this as "no check that I am aware of" rather than a verified negative, and confirm against your version's release notes.
One correction to the premise before you do the math: verify that all three keys actually exist in your version. gitlab_rails['db_pool'] is a long-standing Omnibus attribute, but Workhorse has historically not opened its own PostgreSQL connections (it proxies to Rails over a socket), and Sidekiq's effective pool is typically derived from its concurrency setting rather than a standalone sidekiq['db_pool'] key. Counting a key your version ignores inflates the sum; missing one it does use understates it. Marked for verification against your installed version.
max_connections (the classic "sorry, too many clients already" error in its logs).gitlab.rb — the file only shows what you explicitly set, while defaults also count. sudo gitlab-ctl show-config dumps resolved attributes as JSON where available; otherwise grep the file and consult your version's defaults.max_connections minus headroom. PostgreSQL reserves superuser_reserved_connections (default 3), replication walsenders, and any other clients (Praefect, a Geo tracking database, monitoring) also consume slots. A fixed buffer of roughly 10–20 connections is a common rule of thumb, not a documented GitLab formula.sudo gitlab-psql -c "SHOW max_connections;", watch SELECT count(*) FROM pg_stat_activity; under load, and check PostgreSQL logs for refused connections alongside ActiveRecord connection-timeout errors in the Rails and Sidekiq logs.Wrap gitlab-ctl reconfigure in a read-only guard, or run the same check in CI against your configuration repository:
gitlab-ctl show-config | jq '
((.gitlab_rails.db_pool // 0)
+ (.sidekiq.db_pool // 0)
+ (.gitlab_workhorse.db_pool // 0)) as $pools
| .postgresql.max_connections as $max
| if ($pools + 10) > $max
then error("pools \($pools) + headroom exceed max_connections \($max)")
else "ok: \($pools) pools vs \($max) max_connections"
end'
The // 0 fallbacks make absent keys (such as a Workhorse pool that does not exist) harmless, and a non-zero exit stops the pipeline before an overflowing configuration is applied. On multi-node topologies, run the check per host and aggregate, because every node's pools land on the same server.
Yes, in my view. A warning during gitlab-ctl reconfigure comparing resolved pool attributes against postgresql['max_connections'] fits the validation Omnibus already performs elsewhere and would catch exactly this oversight. Until such a check exists, the wrapper above is the practical substitute, and the gap is worth filing as a feature request on GitLab's issue tracker rather than waiting for it.
Is this a single Omnibus node, or do Rails and Sidekiq run on separate hosts pointing at one PostgreSQL server? That single answer decides whether the sum above is complete or must span hosts with aggregated results. With the topology and GitLab version, the headroom figure can also be tightened to your actual reserve settings.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.