Splunk DB Connect ↔ External Database: diagnosing intermittent connection pool exhaustion

Short answer

When Splunk DB Connect logs Connection is not available during periods of high search concurrency, the most common cause is that the HikariCP pool has reached its maxConnections limit and threads are timing out while waiting for a free connection. A genuine network glitch or database lock will usually surface with a different SQLState or with a lower ActiveConnections count.

1. Production‑safe pool metrics

• JMX – Each pool registers an MXBean named com.zaxxer.hikari:type=pool (HikariPool‑#). The attributes ActiveConnections, IdleConnections, MaxConnections, and ThreadsAwaitingConnection are read‑only and do not require DEBUG logging.
• Undocumented REST endpoint – /services/dbx/dbx2/pool/ on the Splunk management port returns a JSON payload containing the same fields as the JMX bean. The endpoint is available in all recent 3.x releases, but because it is not officially documented, future versions might modify or remove it.

2. Stability of the /services/dbx/dbx2/pool/ endpoint

In practice the endpoint has been stable across the last two major releases of DB Connect (3.1 and 3.2). It is safe to use for dashboards and alerting as long as the script can fall back to JMX if the endpoint is missing or changes. The Splunk support team recommends keeping the fallback in place because the endpoint is not part of the public API contract.

3. Best‑practice approach to separate causes

1. Collect the four pool metrics every 15–30 seconds (via a scheduled search that calls the REST endpoint or via a JMX exporter). Store the values for at least a week.
2. Correlate spikes in ThreadsAwaitingConnection and a steady rise in ActiveConnections that reach MaxConnections with the timestamps of the “Connection is not available” errors. If the pattern matches, the error is almost certainly due to pool saturation.
3. If the metrics show ActiveConnections well below MaxConnections but errors still occur, look for SQLState 08xxx (network reset) or long lock waits on the database side. Check firewall idle‑timeout settings and the database’s wait‑queue or lock‑monitoring tools.
4. When saturation is confirmed, first consider adjusting the maxConnections setting for the specific database input, ensuring the new value stays comfortably below the database server’s own connection limit. If saturation persists, investigate whether scheduled searches can be staggered or if query runtimes can be shortened.

Missing diagnostic detail that would change the recommendation

Could you provide the exact maxConnections value configured for the database input that is throwing the error? Knowing that number lets us determine whether the pool size is appropriate for the observed concurrency level.

Assumptions and uncertainty

This guidance assumes a Splunk Enterprise instance running DB Connect 3.x on a supported JVM. The exact default values for maxConnections and the connection timeout vary by database type and by the specific input configuration, so the recommendation to increase the pool size is tentative until those values are known.