Datadog APM Span Sampling Limits During High Concurrency

When Datadog APM's ingest-side agent buffer reaches capacity during a concurrency spike, spans are dropped according to a priority tier that the agent enforces before any retention filter or custom rule is evaluated. The documented behavior is as follows:

1. spans with sampling.priority = 0 are discarded first
2. spans with sampling.priority = 1 are kept only if they fall within the global sample rate (e.g., if the rate is 10%, roughly 1 in 10 traces at this priority are retained)
3. spans with sampling.priority = 2 are always retained regardless of the global rate

This priority-based dropping occurs on the agent side, meaning that the initial sampling decision happens before any post-ingestion filters apply. Therefore, whether an outlier latency span survives buffer pressure depends entirely on its assigned priority at ingestion.

Regarding the possibility of 100% sampling for requests exceeding a specific latency threshold: Datadog does not provide a native rule condition that evaluates span latency directly. The trace.sample.rules mechanism can force a span to priority 2, but it requires a custom tag (for example, high_latency:true) to be added by application instrumentation when a span's duration exceeds the desired threshold. Without such a tag, the only ways to increase capture of rare high-latency traces are to raise the global sample rate or rely on post-ingestion retention filters. Neither of these guarantees that the initial sampling decision will preserve the outlier span, because the dropping occurs before those stages.

If your application already instruments a latency-based tag, a trace.sample.rules rule can be configured to promote those spans to priority 2, ensuring they are retained even during buffer saturation. If that instrumentation is not in place, the practical next step is to evaluate whether adding a duration-check tag is feasible, or to accept that raising the global sample rate will increase overall trace volume while still subjecting individual spans to probabilistic dropping.

Verification: In Datadog Trace Search, inspect the sampling.priority field on a span; or set DD_TRACE_DEBUG=true to surface agent dropping decisions in the logs. To determine whether a trace.sample.rules approach or a global rate adjustment is the appropriate next step, it helps to know whether your instrumentation currently adds a latency-based tag to spans before they enter the Datadog agent.