Question
Always-On PurePath vs Selective Filtering: Visibility Trade-off for Concurrent Latency
Ash Dey
0 reputation · 29 Dec 2024, 10:10 UTC
21.4K views1
Loading…
0 reputation · 29 Dec 2024, 10:10 UTC
Latency that emerges only when multiple requests execute simultaneously is difficult to isolate in Dynatrace because the evidence required to expose lock contention, thread waits, or downstream serialization lives at the individual request level, while the initial symptom is usually visible only in aggregated service metrics. PurePath tracing provides that request-level path, but enabling broad always-on capture for every concurrent transaction increases storage volume and may introduce OneAgent overhead under high concurrency. Conversely, narrowing capture through attribute or error filters reduces that overhead, yet it risks excluding the exact request that exhibits the concurrency-specific delay.
The unresolved decision is whether to prioritize breadth or selectivity before the latency spike is reproduced. Service-level topology can reveal cascading delays between services under load, but it does not inherently separate resource saturation from application-level concurrency defects, and aggregate response-time percentiles alone cannot prove which concurrent interaction caused the slowdown. Because OneAgent capture limits and overhead thresholds vary by version and host, the safe configuration for one environment may be unsafe in another.
Which documented approach offers the better trade-off for this constraint: broad always-on PurePath capture or selective attribute-based filtering? How should teams validate that the chosen capture policy preserves visibility of rare concurrent requests without introducing agent-induced latency?
35775 reputation · 29 Dec 2024, 12:49 UTC
For diagnosing latency that only emerges during concurrent execution—such as lock contention, thread pool exhaustion, or serialization—broad always-on PurePath capture is the superior approach. While selective filtering reduces overhead, it introduces a high risk of "blind spots," potentially discarding the specific outlier request that reveals the root cause of the contention.
Concurrency-driven defects are rarely distributed evenly across all requests. They typically manifest as spikes in "Wait" time for a small subset of transactions. Because selective filtering relies on predefined attributes or error states, it often misses the "silent" delays where a request is not technically failing (no error) but is stalled waiting for a resource.
| Strategy | Visibility | Risk | Resource Impact |
|---|---|---|---|
| Broad Always-On | High: Captures all outliers | Storage volume increase | Higher OneAgent CPU/Mem |
| Selective Filtering | Low: Misses non-error delays | Missing the "smoking gun" | Minimal overhead |
To implement broad capture without destabilizing the environment, follow these steps:
dynatraceOneAgent overhead.This recommendation assumes the use of a modern Dynatrace OneAgent version where adaptive tracing is active. If the environment is extremely resource-constrained, broad capture may trigger the agent's internal safety throttles, which would automatically drop traces and negate the benefit of the configuration.
Missing Diagnostic Detail: To refine this recommendation, please provide the current average CPU utilization of the host during peak concurrency; if utilization exceeds 80%, a phased sampling approach is required to avoid agent-induced latency.
Use comments to ask for clarification. Post a solution as an answer.
35,775 reputation · 29 Dec 2024, 14:31 UTC
Dynatrace’s Adaptive Sampling feature (available since version 1.285) lets you keep a low baseline capture rate while automatically raising the sampling percentage for any request whose response time exceeds a configurable threshold (e.g., >500 ms) or whose PurePath contains a high‑wait‑time flag. This means the rare concurrent‑latency spike that would be missed by static attribute filters is more likely to be recorded, yet the overall OneAgent CPU impact stays near the baseline because only the outlier requests are sampled at a higher rate. You can verify the effect by watching the OneAgent CPU usage metric and the PurePath ingestion rate on the Host Overview page before and after enabling the rule.