What is the expected latency between log ingestion and search availability in Log Explorer?
0 reputation · 30 Sept 2024, 23:28 UTC
0 reputation · 30 Sept 2024, 23:28 UTC
Datadog processes logs through a distributed indexing system where data first enters a stream before being committed to the searchable index. While the Live Tail feature allows for near real-time visibility by bypassing the indexing process, standard queries in the Log Explorer rely on the completed index state.
In high-volume environments, there is a documented gap between the time a log is ingested and the moment it becomes available for filtered search queries. This behavior can lead to inconsistencies where logs are visible in the raw stream but absent from indexed search results during the same time window.
Which factors primarily influence the duration of this indexing delay, and is there a defined maximum latency window for logs to transition from the stream to the searchable index?
31775 reputation · 01 Oct 2024, 04:16 UTC
For standard Datadog log ingestion, logs typically become searchable in the Log Explorer within a few seconds of ingestion — commonly under roughly 10 seconds under normal load. There is no hard, guaranteed maximum latency window published as an SLA: indexing delay is a best-effort behavior that scales with ingest volume, indexing load, and regional service health. During bursts or heavy shared load, the gap can stretch to tens of seconds or, in some cases, around a minute.
The behavior you describe — visible in the raw stream, absent from search — is a consequence of the two-stage pipeline:
So Live Tail visibility and Log Explorer visibility are not two views of the same state — they read from different points in the pipeline. A log can legitimately exist in one and not the other for a short window.
No contractual maximum is documented for this transition — treat "a few seconds, occasionally tens of seconds" as the expected range, not a guarantee. If you consistently observe delays beyond a minute under normal load, that is outside the typical envelope and worth investigating as a possible throttling, pipeline misconfiguration, or backend issue rather than accepting as normal.
{"marker":"lat-check-20261015","ts":"2026-10-15T02:20:00Z"}.If your organization monitors Datadog usage metrics, check for platform-reported indexing lag under Usage / Logs to corroborate your measurements.
Because search depends on the completed index, monitors that query very recent log windows can miss events still in the indexing buffer. A common mitigation is to evaluate monitors against a window that ends a minute or more in the past, accepting slightly delayed alerts in exchange for not silently skipping unindexed logs.
The figures above reflect generally documented and observed behavior as of this writing; Datadog does not publish a firm latency SLA for indexing, and actual values vary by account, region, and load. If a precise bound matters for your use case (e.g., compliance or incident response timing), confirm current expectations with Datadog support for your plan rather than relying on community-observed numbers.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.