Question
Does APM sampling impact the visibility of intermittent connection pool exhaustion?
Neel Lane
0 reputation · 18 Jun 2024, 06:56 UTC
26K views
Loading…
0 reputation · 18 Jun 2024, 06:56 UTC
Diagnosing intermittent connection pool exhaustion in Java applications requires correlating JVM metrics with specific trace spans. While the Datadog Agent captures database request spans and JVM metrics, there is a potential gap when dealing with transient spikes in connection acquisition time.
If a connection pool exhausts rapidly due to a specific outlier request, the standard APM sampling rate may discard the exact trace responsible for the leak or the long hold time, leaving only the aggregated metric surge in the dashboard.
Which sampling configurations or trace retention settings ensure that these rare, high-latency connection events are captured without incurring excessive costs? Does Datadog provide a mechanism to trigger full trace capture specifically when connection wait times exceed a defined threshold?
31775 reputation · 18 Jun 2024, 07:57 UTC
Yes. If the request that causes a connection‑pool exhaustion spike is not sampled, its trace will be missing from the APM UI, even though the metric surge may still be visible.
When a connection pool exhausts for a short burst, only a few requests experience long wait times. Under a low fixed sampling rate (e.g., 10%) those specific requests may be dropped, so you see only the aggregated metric spike without a trace to pinpoint the root cause.
datadog
javadb.connection.wait_ms exceeds a threshold (e.g., 500 ms).What is the current trace sampling rate or sampling rule configured for your Java service? Knowing this determines whether a simple rate increase or a priority‑sampling rule is the appropriate next step.
Use comments to ask for clarification. Post a solution as an answer.
31,775 reputation · 18 Jun 2024, 08:38 UTC
Datadog’s retention filters operate on the trace level and can keep 100 % of traces that match a span attribute, regardless of the global sampling rate. To capture rare connection‑pool exhaustion spikes you can create a filter that looks for the span attribute db.connection.wait_time (or a custom tag you add) exceeding a threshold—for example, > 500 ms. The filter must be enabled after the trace.search&analytics feature is turned on (agent ≥ 7.30). Unlike priority sampling, which decides per‑trace based on a score, a retention filter guarantees that any trace containing a qualifying span is ingested in full, so you will see the complete request flow in the APM UI even when the overall trace would otherwise be dropped. After adding the filter, restart or reload the Agent and verify in Datadog APM → Trace Search that spans with high wait times appear and are linked to the corresponding surge in the db.connection.wait_time metric.