Datadog Distributed Tracing: Per-service sampling rule precedence
0 reputation · 06 Aug 2020, 16:45 UTC
0 reputation · 06 Aug 2020, 16:45 UTC
Datadog Distributed Tracing allows for trace sampling configuration via the agent. While a global sampling_rate is standard, the sampling_rules array in the agent configuration permits rules based on service names to define specific rates for individual services.
There is uncertainty regarding the evaluation order when overlapping rules exist—specifically when a per-service rule and a global rule both apply to the same trace. It is unclear if the agent consistently prioritizes the most specific match or follows a strict top-down sequence in the configuration file across different Agent versions (e.g., 7.x vs 8.x).
datadog.yaml as the UI does not expose a per-service rule editor.Does the Datadog Agent prioritize per-service sampling rules over global rates regardless of their position in the sampling_rules array? How is the precedence handled when multiple matching rules are defined for a single service?
34775 reputation · 06 Aug 2020, 21:22 UTC
No. The Datadog Agent does not prioritize per-service rules over global rates based on specificity. Treat sampling_rules as an ordered, first-match list: the first rule that matches a trace wins, and later rules are ignored. A per-service rule only takes effect if it appears before any broader rule that also matches.
What is well established is the first-match evaluation model. If your global rate is a separate top-level key such as sampling_rate, it acts as a fallback used only when no rule in sampling_rules matches. If instead the global behavior is implemented as a catch-all rule inside the same array, its position is critical: a catch-all placed above a service-specific rule will shadow that service rule entirely.
The likely explanation for seemingly inconsistent results across versions is not a 7.x-versus-8.x precedence change, but confusion about which layer makes the decision. Tracer-side head sampling can drop a trace before the Agent ever sees it, while Agent-side rules can only filter what they receive. Backend or UI ingestion sampling is a third, separate layer. Mixing these layers makes it look as though rule order is being ignored when in fact the deciding layer never applied the rule.
sampling_rules.Example ordering:
sampling_rules:
- service: "my-service"
sample_rate: 1.0
- service: "*"
sample_rate: 0.1
To narrow the recommendation further, it would help to know whether your global rate is configured as a top-level sampling_rate or as a catch-all rule inside sampling_rules. That distinction determines whether reordering alone is sufficient or whether you also need to move the default out of the array.
Note: Exact key names and Agent recognition of sampling configuration vary by version. Verify against the Datadog documentation for your deployed Agent release before relying on a specific key.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.