Creating a Datadog Log‑Based Metric for Application Error Rates

Problem: scattered error logs hide real‑time error rates

When multiple services write error logs in different formats, aggregating a quick view of how many errors occur per minute requires either custom instrumentation or manual log queries. Both approaches add operational overhead and can miss spikes that happen between polling intervals.

Solution: turn matching log lines into a log‑based metric with a Datadog pipeline

Datadog Logs pipelines can parse incoming log lines, extract attributes (such as level or service), and emit a metric each time a line matches a filter. The resulting metric appears in Metrics Explorer, can be graphed, alerted on, and correlated with APM traces without changing application code.

Worked example: counting level=error lines from a Python service

1. Ensure the Agent is ready

   Run Datadog Agent ≥ 7.x with logs enabled. The Agent must be able to receive logs on its intake endpoint (default http://localhost:10516/v1/input). You need permission to edit the Agent configuration and to restart it if you change logs_enabled: true.

   # Example Agent snippet (yaml)
   logs:
     - type: tcp
       port: 10516
       service: python-app
       source: python
   

2. Create a pipeline in the Datadog UI

   Navigate to Logs → Pipelines → New Pipeline. Choose a name like python-error-pipeline and set the filter to match logs from your service (e.g., service:python-app).

   Add two steps:

   1. Groovy parser to pull out the level field from lines such as:

   level=error service=python-app; msg="Failed to connect to DB"
   

   Use the pattern level=%{word:level} service=%{word:service}; msg=%{greedydata:message}.

   2. Metric generator that fires when level == "error":

      # Metric name: app.error.count
      # Type: count
      # Interval: 1 minute
      # Tags: service, env, version (optional)

3. Send a test log line

   From a host with network access to the Agent, emit a line that matches the parser:

   curl -X POST "http://localhost:10516/v1/input" \
        -H "Content-Type: text/plain" \
        -d "level=error service=python-app; msg="Test error"
   

   You need the ability to open outbound TCP connections to port 10516 on the Agent host.

4. Verify the metric appears

   In Datadog, open Metrics Explorer, search for app.error.count. The metric should show up within the pipeline processing window (typically < 30 seconds). You can confirm the count increments by sending additional error logs and watching the graph rise.

5. Optional: create a monitor

   Set a monitor on app.error.count > 5 for the last 5 minutes to get alerted when error rates spike.

Trade‑offs and limitations

• High cardinality risk: extracting many unique values (e.g., full error messages) as separate metric tags can explode the number of time series, increasing storage cost and slowing queries. Keep tags limited to low‑cardinality fields like service, env, or version.
• Ingestion latency: pipelines run after the Agent receives logs, so near‑real‑time alerts may lag a few seconds. For sub‑second detection you would need custom instrumentation or APM error tracking.
• Dependence on log volume: if you lower the logging level or apply log‑sampling, the metric will under‑count actual errors. Ensure your logging configuration captures all error‑level events you want to monitor.

Actionable closing

Start by adding a simple pipeline that counts level=error lines from one of your services. Verify the metric appears in Metrics Explorer, then extend the pipeline with additional filters (e.g., specific error codes) or combine the metric with APM error spans for richer correlation. Monitor the cardinality of any new tags you introduce, and adjust log sampling if you notice missing counts. This approach gives you a quick, code‑free visibility layer for error rates across your distributed system.