Cloud SQL Proxy connection pool exhaustion and metric visibility
0 reputation · 04 Apr 2021, 13:51 UTC
0 reputation · 04 Apr 2021, 13:51 UTC
The Cloud SQL Proxy manages a per-process connection pool to limit TCP connections to a Cloud SQL instance. While the --max_connections flag allows for pool size adjustments, the proxy does not dynamically synchronize these limits with the instance's internal max_connections setting.
In environments where client libraries (such as JDBC or Go) implement their own pooling layers, a double-layered limit is created. This architecture can lead to intermittent connection refusals when the proxy pool is exhausted, yet the proxy does not natively export real-time pool usage metrics to Cloud Monitoring (Stackdriver).
Given these constraints, how can current pool utilization be monitored without relying solely on retrospective log analysis for "connection pool exhausted" entries? Additionally, what is the recommended strategy for aligning client-side pool sizes with the proxy's --max_connections limit to prevent silent exhaustion?
34775 reputation · 04 Apr 2021, 21:46 UTC
Because the Cloud SQL Proxy does not export internal pool gauges to Cloud Monitoring, you must monitor utilization via client-side wait metrics and OS-level TCP state. To prevent silent exhaustion, client-side pools must be sized as a fraction of the proxy's --max-connections limit, accounting for the number of application processes sharing that proxy.
Since you cannot rely on a native proxy metric, use these two signals to detect pressure before connection refusals occur:
Monitor the connection pool metrics of your application library. A rising number of "waiters" or "pending connections" indicates the proxy pool is full or the instance is saturated.
PendingConnections.DBStats.WaitCount and WaitDuration.numActive against maxActive.The proxy maintains a TCP connection to the instance for every active session in its pool. You can derive current utilization by counting established sockets from the proxy process to the instance port:
# Replace [PID] with the proxy process ID and [PORT] with the instance port
ss -tpn | grep "pid=[PID]" | grep ":[PORT]" | grep ESTAB | wc -l
Divide this count by your configured --max-connections value to get a utilization percentage. This can be scraped by a monitoring agent as a custom gauge.
To prevent exhaustion, align your limits using a top-down approach. Assumption: This assumes a standard deployment where the proxy limit is per-instance.
Calculate the proxy's --max-connections based on the instance's global limit, leaving headroom for maintenance and other proxies:
Proxy Limit = floor((Instance Max Connections - Reserved) * Headroom Factor / Total Proxy Processes)
Divide the proxy limit among the application processes sharing that specific proxy instance:
Client Pool Max = floor((Proxy Limit * Safety Factor) / App Processes per Proxy)
Ensure client pools do not hold idle connections indefinitely, which "pins" the proxy pool. Configure maxLifetime and idleTimeout in your client library to be shorter than the proxy's own timeouts.
Before applying these limits, verify the following:
cloud-sql-proxy --help. Depending on the version, the flag may be --max-connections (hyphen) or --max_connections (underscore).superuser_reserved_connections in Postgres) from the instance total.Missing Diagnostic Detail: To provide a precise sizing formula, please specify how many application processes share a single proxy process. If this is a 1:1 sidecar deployment, the calculation simplifies significantly; if it is a shared standalone proxy, the risk of silent exhaustion is higher.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.