Log Indexing vs. Log Archives for Long-Term Compliance

Primary Decision Factors

The choice between keeping logs indexed in Datadog’s Log Explorer and moving them to low‑cost archives hinges on four observable factors:

• Log value or severity – error, security, or performance‑critical logs have higher operational value.
• Query frequency – logs that are searched multiple times per day benefit from instant indexing.
• Compliance mandates – some regulations require immediate query capability for certain data types.
• Cost sensitivity – indexed storage incurs ingestion and storage fees proportional to volume, while archive storage is orders of magnitude cheaper.

When Rehydration Latency Outweighs Cost Savings

Rehydration pulls raw logs from object storage back into the index, a process that typically takes minutes to hours depending on the requested time window and network bandwidth. The latency penalty becomes a deciding factor when:

• Incident response or forensic analysis must be completed within a strict SLA (e.g., sub‑hourly).
• Regular audits require ad‑hoc queries across large historical ranges, making repeated rehydration impractical.
• Real‑time alerting on archived data is needed, which is impossible without prior rehydration.

Practical Steps to Set the Threshold

1. Classify log sources by severity and typical usage (high‑value/frequent vs low‑value/infrequent).
2. Estimate monthly indexed volume and associated cost using Datadog usage reports.
3. Estimate archive storage cost for the same volume in your chosen object store.
4. Define the maximum acceptable latency for investigations (e.g., 30 minutes, 2 hours).
5. Choose a retention window (e.g., last 14 days) that keeps high‑value/frequent logs indexed; move older logs to archives.
6. Validate by performing a test rehydration of a known archive bucket for a representative time range and measuring the elapsed time.

If your organization requires sub‑hourly log access for investigations, please confirm the maximum acceptable latency; this may shift the threshold toward longer indexing periods.