Constant vs Adaptive Sampling for Intermittent Connection Pool Exhaustion

Question

In a high-throughput service, intermittent connection pool exhaustion produces rare stalls that are difficult to capture without overwhelming the Jaeger backend. Constant sampling at a high probability guarantees those rare traces are recorded, but the fixed overhead adds predictable network and storage cost. Adaptive sampling protects the collector by tuning rates toward a target samples-per-second budget, yet its dynamic reduction can skip the low-frequency traces that expose pool waiting.
The decision is further constrained by instrumentation scope: Jaeger only reflects pool waiting when the connection pool library emits explicit spans, so the sampling strategy operates on an already partial signal. Remote sampling adds another variable, allowing the collector to push configuration changes without client restarts but introducing a delay between policy updates and client behavior.
Which sampling approach best preserves visibility into intermittent pool exhaustion under a fixed collector budget? How should the constant probability or adaptive target be chosen when stall frequency is unknown and the pool is instrumented? Does remote sampling configuration lag matter for short-lived incidents?

Tasadduq Burney · Answer

Recommended Sampling Approach
For intermittent connection pool exhaustion, constant sampling is the superior choice for preserving visibility. Adaptive sampling typically optimizes for a total throughput budget, which often results in the suppression of low-frequency "outlier" traces—the exact traces needed to diagnose rare pool stalls.

Analysis of Sampling Strategies
The trade-off between these strategies depends on whether you prioritize system stability or the capture of "long-tail" latency events:

Constant Sampling: Provides a statistically consistent window into all requests. If a pool stall occurs in 0.1% of requests and you sample at 1%, you are mathematically guaranteed to capture a representative sample of those stalls over time, provided the collector can handle the volume.
  Adaptive Sampling: Dynamically reduces sampling rates during traffic spikes to protect the backend. Because pool exhaustion often coincides with high-load bursts, adaptive sampling may aggressively throttle the very traces that contain the pool-wait spans.

Configuration Guidelines
When stall frequency is unknown but the pool is instrumented, use the following approach to determine your probability target:

Establish a Baseline: Start with a very low constant probability (e.g., 0.001) to verify that the collector is receiving spans.
  Stress Test: In a staging environment, simulate pool exhaustion (e.g., by reducing the max pool size or introducing artificial latency).
  Calculate Capture Rate: Monitor the ratio of pool-wait spans captured versus total requests. Increase the constant probability until the number of captured stalls per incident is sufficient for root-cause analysis without triggering collector drop rates.
  Monitor Collector Health: Use the Jaeger collector metrics to ensure the spans_dropped count remains at zero. The maximum sustainable constant probability is the point just before the collector begins dropping spans.

Remote Sampling and Incident Lag
Remote sampling configuration lag does matter for short-lived incidents. Because there is a delay between updating the policy on the collector and the client picking up the new probability, a spike that lasts only a few seconds may be entirely missed if you attempt to "turn up" sampling reactively. To capture transient spikes, the sampling rate must be set to a sufficient constant level before the incident occurs.

Diagnostic Requirement
To refine this recommendation, please specify: Is the connection pool library emitting spans for the "wait" duration specifically, or only for the total checkout time? If the library only tracks total time, a higher sampling rate is required to differentiate between network latency and pool exhaustion.

Constant vs Adaptive Sampling for Intermittent Connection Pool Exhaustion

1 answer

Recommended Sampling Approach

Analysis of Sampling Strategies

Configuration Guidelines

Remote Sampling and Incident Lag

Diagnostic Requirement

Your perspective belongs here.

0 question comments