Controlling Observability Costs with Datadog Log-to-Metrics

When a microservices deployment misbehaves, the reflex is to log everything: every request, every retry, every provider response. That instinct is right for forensics and expensive for everything else. A payment service emitting one structured log line per transaction can produce tens of millions of indexed lines per day, and most of them are only ever consumed by an aggregation — an error rate, a latency percentile, a count per provider. You end up paying indexed-log prices to store data whose only real use is a number on a dashboard.

Datadog's Log-to-Metrics feature addresses exactly this gap. It generates custom metrics directly from your log stream at ingestion time — no application code changes, no additional agents — so you can keep the trends and alerts you actually use as metrics, then shorten how long the raw lines stay indexed. This post walks through when that trade makes sense, a concrete configuration, and the cardinality pitfalls that turn a cost optimization into a cost problem.

The high-cardinality trap

Cardinality here means the number of unique tag combinations a datapoint can have. Per-request logs are inherently high-cardinality: they carry user IDs, request IDs, container IDs. Keeping every unique event as a searchable log is costly; emitting every unique event as a custom metric is worse, because each unique tag combination becomes a billable time series against your custom-metrics quota.

Log-to-Metrics is the middle path. You define a filter over the log stream, and Datadog aggregates matching logs into a metric regardless of what happens to the raw lines afterward. The metric keeps the low-cardinality dimensions you choose — provider, region, status code — and discards the per-event noise.

What you can generate

Each configuration produces one of three metric types from matching logs:

Metric type	Behavior	Typical use
Count	Increments on every matching log	Failed logins, 404s, retries
Gauge	Last extracted value in the interval	Queue depth or memory reported in a log line
Distribution	Tracks the distribution of an extracted value	Request latency in milliseconds

Values come from log attributes (facet paths like @duration_ms) or from a regex applied to the message. Group-by tags determine the time series the metric is split into.

Worked example: payment latency without 30-day retention

Suppose a payment service logs a JSON line per completed transaction:

{"timestamp":"2026-10-13T19:43:11Z","level":"info","message":"payment_processed","provider":"stripe","duration_ms":450,"status":"success"}

To track latency per provider without keeping every line for a month, create a generate-metrics configuration in the Datadog UI under Logs → Generate Metrics (labeled Log-to-Metrics in some plan versions — the naming has shifted, so check your current UI). This requires a role with write access to log configuration; Datadog Admin roles include it by default.

• Filter: service:payment-service @status:success
• Metric name: payment.processing.latency
• Type: Distribution
• Group by: @provider
• Value to extract: @duration_ms

Once active, the metric aggregates every matching log. You can chart p50/p99 per provider on a dashboard, wire it into an SLO, and cut retention for these log lines to a few days — or route them to an archive instead of an index. The trend data lives in the metrics platform, which retains metrics far longer than a short log index window.

The trade-offs

Tag cardinality is the big one. Every unique combination of group-by tags creates a distinct custom metric time series. Group by @provider and you get a handful of series; group by @user_id and you generate one per user, which can exhaust your monthly custom-metrics quota quickly. Stick to low-cardinality dimensions and review the Usage page after enabling anything new.

There is ingestion latency. Generated metrics appear in the metrics platform minutes after the logs arrive, not instantly. This feature suits dashboards, alerting, and SLOs — not real-time controls like circuit breakers.

Regex has a processing cost. On high-volume streams, broad or backtracking-heavy patterns add load to log processing. Prefer extracting from structured attributes (the @field paths) over regexing raw messages whenever your logs are already parsed.

You lose per-event drill-down. Once raw logs age out of the index, a spike on the metric can't be traced back to individual lines. Keep a longer retention — or an archive with rehydration — for the log patterns you may need to investigate.

Verifying the pipeline

1. Confirm the configuration shows as active on the generate-metrics page in the Datadog UI.
2. In Metrics Explorer, compare the generated metric's count against a raw log query with the same filter over the same window. They should align closely; small gaps in the first minutes are expected from processing delay.
3. Check the Usage page for unexpected growth in custom metrics — a cardinality mistake usually shows up here within hours.

If you're starting from an oversized log bill, the practical move is incremental: identify your three to five highest-volume log patterns, generate metrics for the ones feeding dashboards and alerts, verify the counts match, then shorten retention for those indexes. Keep raw logs only where per-event forensics genuinely matters. Because quotas, pricing, and UI naming change across Datadog plans and releases, confirm current limits in Datadog's documentation before committing to a rollout.