Cached HBase connection eviction vs. Phoenix Query Server centralization: trading pool controllability for latency under intermittent exhaustion

Confirmed Facts

• Phoenix thick driver caches one HBase Connection instance per client configuration; closing a JDBC connection does not immediately release the underlying region-server resources.
• HBase's hbase.connection.idle.timeout governs when idle connections are evicted, but the idle clock resets on any connection activity, making eviction timing nondeterministic from the client perspective.
• The thin driver routes all queries through Phoenix Query Server (PQS), centralizing HBase connection management and eliminating per-client connection caching.
• Phoenix exposes metrics such as HBaseConnectionPoolExhausted and QueryServerPoolExhausted (naming varies by release) to help distinguish exhaustion origin.

Likely Explanation

For eviction to become deterministic from the client side, the HBase idle timeout must be set low enough that background traffic does not repeatedly reset the idle clock. In practice, this requires:

1. hbase.connection.idle.timeout configured in hbase-site.xml to a value (e.g., 30–60 seconds) that is shorter than the typical background activity interval.
2. Disabling or reducing background operations (e.g., periodic meta scans, heartbeat RPCs) that might reset the idle timer.
3. Using Phoenix JDBC properties such as PhoenixConnectionLeaseMonitoringInterval to log connection lease durations and verify eviction behavior.

However, HBase does not provide a client-side trigger for forced eviction; the only reliable mechanism is waiting for the timeout to elapse without activity. Therefore, deterministic eviction from the client is only achievable when the system is idle long enough for the configured timeout to expire.

Migrating to Thin Driver via Phoenix Query Server

Enabling PQS eliminates per-client HBase connection caching, which removes the primary cause of intermittent client-side pool exhaustion. Instead, all clients share a single PQS instance (or cluster) that manages its own connection pool to HBase. This shifts the exhaustion risk to the PQS tier, where pool saturation becomes a function of total query concurrency rather than per-client connection leaks.

Metrics to Distinguish Exhaustion Source

For a deployed Phoenix release, the following metrics differentiate exhaustion origin:

• PQS-tier exhaustion: QueryServerPoolExhausted counter increments when PQS cannot allocate a connection to HBase. This metric appears in PQS JMX endpoints or Prometheus scrapes.
• Client-tier exhaustion (thick driver): HBaseConnectionPoolExhausted or equivalent counters increment when the cached HBase connection cannot be reused due to lease conflicts or timeout issues. These are exposed via the Phoenix JDBC driver's metrics registry.

Verification requires checking that both metrics are enabled in hbase-site.xml and phoenix-site.xml, then monitoring them under load. If only QueryServerPoolExhausted increments, the exhaustion is centralized; if HBaseConnectionPoolExhausted increments without PQS involvement, the thick driver's caching behavior is the culprit.

Recommended Diagnostic Step

Confirm the exact HBase and Phoenix versions in use, as metric names and timeout defaults have changed across releases. For example, hbase.connection.idle.timeout default values differ between HBase 1.x, 2.x, and 3.x, and Phoenix metric exposure evolved in versions 4.14, 5.0, and 5.1+.