From Log Spam to Signal: Using Datadog Log-Based Metrics for Burn-Rate SLOs

At high log volume, a monitor built on a raw log search is usually the wrong tool. Every evaluation scans matching logs, the query gets slower as traffic grows, and grouping by fields like http.url or error.message creates thousands of unique combinations that make the alert both noisy and expensive. The practical alternative is to derive a small set of log-based metrics at ingestion—aggregate counts and distributions computed from log events—and alert on those instead. Raw logs remain available for investigation, but your monitors watch a bounded, low-cardinality signal.

Why raw log monitors degrade under load

Log search is optimized for forensic queries, not for repeated, high-frequency evaluation. When you alert on a log search, each monitor run re-scans the log stream. As the service emits more lines, the query takes longer and costs more. Worse, raw log fields are naturally high-cardinality: a single endpoint with query parameters produces a new URL string per request, and stack traces vary per failure. Grouping by those fields explodes the number of series you are trying to chart or alert on.

A log-based metric collapses that stream at ingestion. Datadog computes the aggregate as logs flow through the pipeline, so the monitor evaluates a metric time series rather than a search. The result is faster, cheaper, and far more stable to alert on.

Choosing dimensions that stay bounded

The cardinality of a log-based metric is determined by its metric name plus the tags you select. Every unique tag combination becomes a separate billable custom metric time series. Include only dimensions that are stable and bounded:

• service and env—core deployment context.
• status class—for example, a coarse bucket such as 5xx or 4xx rather than every HTTP status code.
• endpoint—only if the route template is bounded (e.g., /checkout), not the full URL with IDs.

Never tag with unbounded values: user IDs, session tokens, trace IDs, full URLs, or raw error messages. They will multiply your custom-metric count and make monitors impossible to reason about.

Worked example: checkout errors and latency

Suppose a checkout service emits structured application logs with fields for HTTP status and request duration. You want two signals: an error ratio and a latency distribution.

Metric 1—5xx count. Create a log-based metric that counts log events where the status code is 5xx. Group it by service and env. This gives you a clean numerator for an error budget.

Metric 2—request duration. Create a distribution metric from the duration field on checkout requests, again grouped by service and env. A distribution preserves percentile information, which a simple count cannot.

Before the metric processor runs, use pipeline filters to exclude health checks and static asset requests. If those noisy events reach the metric step, they dilute the signal.

With both metrics in place, define a service-level objective (SLO) such as “99.9% of checkout requests succeed over a rolling 30-day window.” Rather than alerting on a single static threshold, configure a multi-window burn-rate alert: a fast window catches rapid error spikes, and a slow window catches sustained degradation. This mirrors how SLOs are meant to be operated and avoids paging on transient blips.

Trade-offs: ingestion gaps and pipeline order

Log-based metrics inherit every weakness of your log pipeline. If logs are dropped by the agent, sampled, or fail a parser before the metric processor runs, the metric silently undercounts. A monitor can look healthy because the errors never became metric points. Pipeline behavior—including Grok parsing rules, OpenTelemetry log handling, and retention settings—varies by organization and changes over time, so validate current behavior in your own tenant rather than assuming another team’s configuration applies.

Processor order matters. Sensitive-data redaction and exclusion filters must execute before the metric is computed; otherwise you risk tagging metrics with scrubbed fields or counting events you meant to drop. In the Datadog UI, open the pipeline for the service and confirm the processor order. Use the pipeline preview on sample logs to verify that the status and duration attributes exist before the metric step.

Cost is the other trade-off. Custom metrics are billed by the number of unique time series. Adding one high-cardinality tag can increase your bill by an order of magnitude. Review the metric in Metrics Summary or the custom metrics view after a day of real traffic to confirm the tag combinations remain bounded.

How to verify before you rely on it

1. In a non-production service, create a temporary log-based metric with the tags you plan to use.
2. Send synthetic logs with known 5xx counts and known durations. Compare the values in Metrics Explorer against your expected numbers.
3. Inspect the pipeline processor order and use the preview tool to confirm attributes are parsed before the metric processor.
4. Trigger the monitor with a controlled failure, or temporarily lower the threshold, and confirm routing and recovery behave as expected.
5. After a day of real traffic, check Metrics Summary for the new metric names and tag combinations. If cardinality is higher than planned, remove tags before expanding.

Start narrow, then expand

Pick one high-volume service. Exclude health checks and static assets. Define a request/error SLO. Create the two log-based metrics—one count, one distribution—and attach a burn-rate monitor. Verify counts against synthetic data, confirm cardinality stays flat, and only then repeat the pattern for other services. The goal is not to replace logs; it is to stop alerting on them.