Question
Cloud SQL Auth Proxy v2.5.0 max_connections flag: unbounded queueing versus instance max_connections boundary
Mira Lane
0 reputation · 21 Apr 2024, 10:31 UTC
54.1K views1
Loading…
0 reputation · 21 Apr 2024, 10:31 UTC
Cloud SQL Auth Proxy v2.5.0 introduced a proxy-level max_connections flag that operates independently of the Cloud SQL instance's max_connections database flag. When the proxy flag is unset (the default), the proxy queues incoming application connections indefinitely instead of failing fast. This queues masks pool exhaustion as latency spikes rather than explicit connection errors, while the instance enforces its own max_connections limit derived from RAM tier (approximately 100 connections per GB for PostgreSQL 14+).
In environments where Cloud Run or Cloud Functions scale proxy instances independently, each proxy replica maintains its own local pool. Total connections equal proxy_instances × pool_size_per_proxy, not a single shared pool. The proxy exposes no per-pool metrics (idle, used, waiting) via standard endpoints, forcing operators to correlate proxy logs (New connection for at INFO level) with the Cloud Monitoring metric database/postgresql/num_backends to detect saturation. Meanwhile, superuser_reserved_connections (default 3) further reduces the effective pool for non-superuser roles, and this flag is not configurable in Cloud SQL.
Establish a reliable diagnostic and configuration pattern that prevents intermittent FATAL: remaining connection slots are reserved for non-replication superuser connections errors by making proxy-side queueing visible and bounding it against the instance connection limit.
FATAL errors, given the absence of exposed pool metrics?max_connections flag across N Cloud Run instances against a fixed instance max_connections limit?max_connections flag change the failure mode from latency spikes to explicit connection rejections, and how does that interact with application-side pool timeouts (e.g., HikariCP connectionTimeout)?35775 reputation · 21 Apr 2024, 18:50 UTC
With no exposed pool metrics, operators must correlate three signals:
google
postgresqlNew connection for lines per proxy instance indicates incoming demand exceeding forwarding capacity.database/postgresql/num_backends approaching the instance max_connections value (run SHOW max_connections; on the instance to confirm).connectionTimeout waits or increasing pool acquisition times signal that the proxy queue is draining slowly.Set up a log-based metric counting New connection for per proxy instance (via resource.type="cloud_run_revision" and labels.instance_id) and alert when the rate exceeds proxy_max_connections / expected_acquisition_time.
proxy_max_connections across N Cloud Run instancesUse the conservative bound:
proxy_max_connections ≤ floor((instance_max_connections - superuser_reserved_connections - safety_margin) / max_proxy_instances)Where:
instance_max_connections = value from SHOW max_connections; (≈ 100 per GB RAM for PostgreSQL 14+)superuser_reserved_connections = 3 (fixed in Cloud SQL, not configurable)safety_margin = 10–20% of instance_max_connections for Cloud SQL internal connections and uneven scalingmax_proxy_instances = Cloud Run max_instances setting (must be known and stable)Example: 5 GB instance → instance_max_connections=500, max_proxy_instances=20, 10% margin → proxy_max_connections ≤ floor((500 - 3 - 50) / 20) = 22.
proxy_max_connections enabledYes. With the flag unset (default): proxy queues unboundedly → application sees latency spikes (HikariCP waits up to connectionTimeout) → instance eventually returns FATAL: remaining connection slots are reserved for non-replication superuser connections.
With the flag set: proxy refuses new connections immediately (TCP RST / connection refused) → application receives SQLException: Connection refused or HikariCP PoolEntryCreator timeout within milliseconds → saturation becomes visible as errors, not latency.
HikariCP tuning: set connectionTimeout to 1–2 seconds (typical proxy acceptance latency + network RTT). A high timeout (default 30s) masks the fast-fail benefit by retrying against a refusing proxy.
cloud-sql-proxy --version ≥ 2.5.0.SHOW max_connections; on the Cloud SQL instance; confirm superuser_reserved_connections=3 via SHOW superuser_reserved_connections;.max_instances to a known ceiling (e.g., 20). This value must not change without re-evaluation.proxy_max_connections using the formula above. Deploy via Cloud Run env var or command arg: --max-connections=22.New connection for per instance_id.database/postgresql/num_backends vs. instance_max_connections - 3 - margin, alongside the proxy connection rate metric.max_instances, burst connections, verify proxy rejections appear in client logs before instance num_backends hits the safety boundary.What is your Cloud Run service's configured max_instances value? The sizing formula requires a fixed ceiling; if autoscaling is unbounded or the limit is unknown, the formula cannot be applied safely.
Use comments to ask for clarification. Post a solution as an answer.
6,690 reputation · 21 Apr 2024, 22:23 UTC
Enabling the proxy max_connections flag shifts the failure mode from indefinite latency spikes (default queueing) to explicit connection rejections when the configured proxy limit is hit. This allows application-side pool timeouts, like HikariCP's connectionTimeout, to trigger predictably before the Cloud SQL instance's hard limit is breached.