Question
Peak headroom or request throttling: sizing the Cassandra driver pool for bursts
Sky Indigo
0 reputation · 10 Jun 2024, 02:50 UTC
90.2K views1
Loading…
0 reputation · 10 Jun 2024, 02:50 UTC
The DataStax Java Driver 4 sizes each node's connection pool statically at session start via PoolingOptions — maxConnectionsPerHost together with maxRequestsPerConnection. Nothing resizes the pool in response to load, so bursts that exceed the configured in-flight capacity end up rejected or queued depending on the driver version — the classic intermittent pool-exhaustion pattern.
Two documented levers pull in opposite directions. Size the pool for the peak: raise maxConnectionsPerHost so bursts always find headroom, accepting idle sockets and more connections per node. Or keep a modest pool and cap in-flight requests with the driver's request throttler (a concurrency-limiting throttler is one documented option), so bursts queue briefly instead of exhausting the pool, at the cost of tail latency.
Constraints worth weighing: the workload is bursty rather than steady; pool defaults and busy-pool behavior differ between driver 3.x and 4.x and need verification against the version in use; and node-side connection limits bound how far the headroom lever can go.
For intermittent, burst-driven exhaustion, which trade-off fits better — peak-sized headroom in the static pool, or throttled concurrency against a small pool? And which driver metrics (in-flight requests per connection, pool size, throttled-request counts) most reliably reveal which lever is the real bottleneck?
30275 reputation · 10 Jun 2024, 11:55 UTC
For a genuinely bursty workload, keep the connection pool modest and enable the DataStax Java Driver’s request throttler (concurrency‑limiting mode). This converts pool‑exhaustion failures into bounded queueing, trading a modest rise in tail latency for far fewer idle sockets.
cassandraPoolingOptions (or basic.connection.pool.* and max-requests-per-connection). No dynamic resizing occurs.max-requests-per-connection (1024); driver 3.x used multiple connections per host and threw BusyPoolException when exhausted.maxConnectionsPerHost × maxRequestsPerConnection. Anything beyond this must be throttled or fail.native_transport_max_concurrent_connections) bound how far the headroom lever can go.session.getContext().getConfig() and log basic.connection.pool.local-size, basic.connection.pool.remote-size, basic.request.throttler.mode and basic.request.throttler.max-concurrent-requests.in-flight-requests per connection approaching max-requests-per-connection indicates pool saturation.throttler.queue-size and throttler.delay show how much burst is being absorbed.throttler.rejected-count (should be zero when throttling works) vs pool.exhausted-count.max-requests-per-connection or pool size safely.To finalize the recommendation, please confirm whether the Cassandra cluster’s native‑transport connection limit is near utilization during bursts; if it is, headroom gains are unsafe and throttling is the only viable lever.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.