Splunk DB Connect ↔ External Database: diagnosing intermittent connection pool exhaustion

Goal

Identify the root cause of sporadic “Connection is not available” errors reported by Splunk DB Connect while querying an external database.

Constraints

Errors appear only under high search concurrency; logs show timeouts but lack clarity on whether the pool is saturated, a network hiccup, or a database lock.

Unresolved Behavior

DB Connect’s pool metrics are exposed only via JMX or an undocumented REST endpoint; it is unclear how reliable these are for production monitoring.

Questions

1. Which pool‑state metrics can be safely exposed in a production Splunk instance without enabling DEBUG logging?
2. Is the undocumented /services/dbx/dbx2/pool/ endpoint stable enough for automation?
3. What best‑practice approach exists to distinguish between load‑driven exhaustion and transient network or lock issues?