Diagnosing Datadog Ingest Pipeline Throttling: A Practical Step‑by‑Step Guide

Recognizable Condition

The Ingest Rate metric remains at 100 % or Datadog logs indicate a delay longer than 5 minutes, signaling that the pipeline is throttling incoming data.

Cause Overview

Category	Typical Trigger
Network congestion	High upstream traffic or unstable network links.
Excessive tag cardinality	Too many unique tag values per event.
Mis‑configured batch size	Batch size too small for the volume, causing frequent flushes.
Upstream log volume spike	Sudden increase in log generation rate.
Outdated Datadog Agent	Older Agent versions lack newer throttling safeguards.

Ordered Checks

1. Open the Agent status page in Datadog and confirm the Ingest Rate metric is at 100 % for at least 10 minutes.
2. Run `ddagent status` on the host and verify the queue depth and current batch size match expected values.
3. Inspect recent log entries for high tag cardinality; calculate tag entropy if needed.
4. Check the Agent version (`ddagent --version`) and compare against the latest supported release.

Fixes Tied to Findings

• Increase the pipeline batch size in `datadog.yaml` (e.g., `batch_size: 2000`) and restart the Agent.
• Enable compression (`compress: true`) to reduce payload size.
• Prune or consolidate high‑cardinality tags in the log shipper configuration.
• Upgrade the Datadog Agent to version 8.0 or newer, which exposes queue depth metrics.
• Adjust the upstream log shipping rate to smooth spikes before they reach the Agent.

Escalation Criteria

If after applying the fixes the Ingest Rate remains at 100 % for more than 30 minutes, or if queue depth continues to grow without bound, escalate to Datadog support with the following data:

• Full Agent status output (`ddagent status`).
• Relevant log excerpts showing tag cardinality and batch size settings.
• Network latency metrics from the host.