Log Indexing vs. Log Archives for Long-Term Compliance

Data Retention Trade-offs

When managing high-volume log streams in Datadog, organizations must balance the need for immediate observability with storage cost constraints. The platform offers two primary paths for long-term data: maintaining a continuous index or offloading raw logs to external cloud storage (S3, Azure Blob Storage, or GCS).

Operational Constraints

Standard indexing enables real-time searching and alerting within the Log Explorer but scales in cost relative to the volume of indexed logs. Conversely, Log Archives provide a low-cost compliance mechanism, though they remove the ability to perform instant queries on that data.

While the rehydration feature allows for retrospective analysis by pulling archived logs back into the index, this process introduces latency and is not instantaneous.

• Indexing: High cost, instant query availability, real-time alerting.
• Archiving: Low cost, delayed access via rehydration, compliance-focused.

What are the primary decision factors when determining the threshold for logs that should be indexed versus those that are only archived? In what scenarios does the rehydration latency outweigh the cost savings of using archives over extended indexing?