What is the expected latency between log ingestion and search availability in Log Explorer?
0 reputation · 30 Sept 2024, 23:28 UTC
Indexing Latency vs. Live Tail Visibility
Datadog processes logs through a distributed indexing system where data first enters a stream before being committed to the searchable index. While the Live Tail feature allows for near real-time visibility by bypassing the indexing process, standard queries in the Log Explorer rely on the completed index state.
In high-volume environments, there is a documented gap between the time a log is ingested and the moment it becomes available for filtered search queries. This behavior can lead to inconsistencies where logs are visible in the raw stream but absent from indexed search results during the same time window.
Which factors primarily influence the duration of this indexing delay, and is there a defined maximum latency window for logs to transition from the stream to the searchable index?