Does Datadog Agent keep priority-1 APM traces after reaching the intake rate limit?

Question

Goal: determine whether Datadog APM retains a tracer-marked priority-1 trace when the Agent APM intake rate limit has already been reached. Tracer sampling priority and Agent intake rate limiting are documented as separate controls, and public documentation does not clearly state which takes precedence when both apply.

Uncertainty: the outcome may depend on the Datadog Agent major version, the language tracer, and the APM receiver configuration in datadog.yaml. Exact keys and defaults should be treated as version-sensitive until confirmed against the installed Agent release notes.

Does the Agent drop priority-1 traces once the APM intake rate limit is exceeded, or does tracer sampling priority override the Agent limit?
Which configuration key, if any, controls this precedence, and does it vary by Datadog Agent major version?
What is the documented retention behavior for priority-1 spans when the Agent is actively rate-limiting APM intake?

Tasadduq Burney · Answer

Direct answer
Datadog public documentation does not confirm that priority-1 APM traces are retained after the Datadog Agent APM intake rate limit is reached. Tracer sampling priority and Agent intake rate limiting are separate controls, and no version-agnostic rule states that priority-1 overrides an Agent-level intake drop. Whether a priority-1 trace survives depends on the installed Agent major version and its release-specific APM receiver behavior.
Confirmed facts

Datadog language tracers support sampling priorities; priority 1 marks a span as user-selected for retention.
The Datadog Agent applies APM intake rate limiting through the apm_config section of datadog.yaml.
Tracer priority is decided before spans leave the application; Agent rate limiting is applied when the Agent receives spans.
Public documentation does not clearly establish which control takes precedence when both apply to the same span.

Likely explanation
Agent intake rate limiting acts as receiver-side backpressure. Once the configured intake rate is exceeded, the Agent may drop excess spans to protect itself. Because this occurs after the tracer has already marked the span, priority-1 marking alone is not documented to prevent an Agent-side drop during saturation. This is likely behavior, not a confirmed guarantee for every Agent release.
Verification steps for this case

Identify the installed Datadog Agent major version; apm_config keys and precedence rules differ by major version.
Inspect datadog.yaml under apm_config for APM intake rate-limit settings on that Agent.
Review the Agent log for APM intake rate-limit or span-drop events while a priority-1 trace is submitted.
Check the release notes or changelog for the installed Agent version for any priority-preserving rate-limit behavior.

Missing diagnostic detail
Provide the Datadog Agent major version running on the host. It determines which apm_config keys to inspect and whether a priority-preserving exception exists for that release.

Does Datadog Agent keep priority-1 APM traces after reaching the intake rate limit?

1 answer

Direct answer

Confirmed facts

Likely explanation

Verification steps for this case

Missing diagnostic detail

Your perspective belongs here.

0 question comments