Internal Pool Scaling vs. External Pooler Deployment: Choosing the Right Approach for ArgoCD Connection Exhaustion

Answer the question first

Start by exposing per‑pod pool usage, calculate a safe internal pool size, tune ArgoCD via argocd-cm, and only add an external pooler (Redis Cluster or PgBouncer) if the tuned internal size still approaches backend limits.

Likely explanation

ArgoCD creates a separate Redis and PostgreSQL connection pool inside each pod. During sync waves many pods open connections simultaneously, which can exceed the backend’s maxclients (Redis) or max_connections (PostgreSQL) even though each pod’s pool looks healthy.

Confirmed facts

• ArgoCD exposes pool size via ConfigMap keys: redis.connectionPoolSize and db.pool.maxOpenConnections.
• No native metrics are emitted for per‑pod pool utilization.
• PgBouncer in transaction‑pool mode discards prepared statements on each connection release, causing “prepared statement not found” errors unless the application re‑prepares them.

Steps to add instrumentation

1. Deploy a side‑car exporter (or use kube-state-metrics with a custom query) that reads:
• Go‑Redis PoolStats via redis.Client.PoolStats() → expose argocd_redis_pool_connections_used and argocd_redis_pool_capacity.
• PostgreSQL pg_stat_activity via a simple SQL query (SELECT count(*) FROM pg_stat_activity WHERE application_name = 'argocd') → expose argocd_pg_connections_used.
2. Expose the metrics on :8080/metrics and scrape with Prometheus.
3. Create a Grafana panel (or use Prometheus directly) with:
• sum by (pod) (argocd_redis_pool_connections_used)
• sum by (pod) (argocd_pg_connections_used)
4. Alert when usage > 80% of the configured pool size.

Calculating the optimal internal pool size

Let:

• W = expected concurrent sync waves (peak number of ArgoCD repo‑syncs running at once).
• C = average connections each wave opens to Redis/PostgreSQL (measure from a load test or from pg_stat_activity during a sync).
• P = number of ArgoCD pods (replicas).
• M = backend‑wide limit (maxclients for Redis, max_connections for PostgreSQL).
• S = safety margin (typically 0.2‑0.3).

Formula:

optimal_per_pod_pool = ceil( (W * C) / P ) * (1 + S)

Then verify:

optimal_per_pod_pool * P < M

If the inequality fails, reduce W (throttle sync concurrency) or increase pod count P before raising the pool size.

When to add an external pooler

If after tuning the internal pool you still observe:

• Backend connection counts approaching M, or
• Frequent “pool exhausted” logs despite optimal_per_pod_pool being far below the configured size.

Deploy an external pooler:

• Redis: use Redis Cluster or a side‑car redis-proxy (e.g., twemproxy) to shard connections.
• PostgreSQL: deploy PgBouncer in session pooling mode (not transaction) to preserve prepared statements.

PgBouncer configuration to avoid breaking prepared statements

1. Set pool_mode = session in pgbouncer.ini.
2. Ensure server_reset_query = DISCARD ALL (default in session mode) so that temporary state is cleared without discarding prepared statements.
3. In ArgoCD, disable persistent prepared statements if you must use transaction mode: set db.connectionParams = ?prepareThreshold=0 (or equivalent) and rely on PgBouncer to reset.
4. If you keep transaction mode, add a post‑connect hook in ArgoCD (via a custom init container) that issues PREPARE statements after each connection is taken from the pool.

Missing diagnostic detail

To confirm whether internal tuning suffices, I need the current backend limits: maxclients for Redis and max_connections for PostgreSQL. Please provide those values so I can verify the inequality optimal_per_pod_pool * P < M and decide if an external pooler is required.