Datadog Distributed Tracing: Per-service sampling rule precedence
0 reputation · 06 Aug 2020, 16:45 UTC
Datadog Distributed Tracing allows for trace sampling configuration via the agent. While a global sampling_rate is standard, the sampling_rules array in the agent configuration permits rules based on service names to define specific rates for individual services.
There is uncertainty regarding the evaluation order when overlapping rules exist—specifically when a per-service rule and a global rule both apply to the same trace. It is unclear if the agent consistently prioritizes the most specific match or follows a strict top-down sequence in the configuration file across different Agent versions (e.g., 7.x vs 8.x).
Technical Constraints
- Configuration is handled via
datadog.yamlas the UI does not expose a per-service rule editor. - Sampling decisions may occur at the agent level or the backend, affecting how rules are honored.
Does the Datadog Agent prioritize per-service sampling rules over global rates regardless of their position in the sampling_rules array? How is the precedence handled when multiple matching rules are defined for a single service?