Choosing Between Static Thresholds and Baselines for New Relic APM Alerts

Problem Statement

In New Relic APM you need to decide how to trigger alerts on production metrics. Two supported condition types exist:

• Static threshold – a fixed numeric value you set.
• Baseline – New Relic compares current values to a statistical baseline built from historical data.

The decision hinges on the nature of the metric, the required compliance level, and operational noise tolerance.

Decision Context & Constraints

Use this guide when:

• You have a clear Service Level Agreement (SLA) that defines hard limits.
• Seasonal or diurnal patterns strongly influence the metric.
• You want to reduce alert fatigue while still catching true degradations.
• Automation via Terraform or the NerdGraph API is required.

Key constraints to consider:

• Baseline conditions need a minimum of 2–4 weeks of metric history (varies by account tier).
• Static thresholds are easier to version‑control and audit.
• Both condition types can coexist in the same policy with independent severities.
• Baseline alerts may suppress known seasonal spikes but can miss slow, steady degradations.

Comparison Table

Feature	Static Threshold	Baseline
Definition	Fixed numeric limit (e.g., 2000 ms)	Statistical baseline derived from historical data
Data Required	None beyond the current value	Several weeks of historical data
Alert Sensitivity	High – any crossing triggers	Adaptive – only significant deviations trigger
False‑Positive Rate	Higher during normal variance	Lower for expected patterns
Compliance Fit	Best for hard SLAs with fixed limits	Better for metrics with natural seasonality
Version‑Control	Easy – store threshold value in IaC	Harder – baseline is learned, not stored
Setup Complexity	Minimal – one value to set	Requires sufficient history and preview
Automation	Supported via Terraform newrelic_alert_condition	Supported via Terraform with baseline block
Alert Fatigue	Potentially high if metric fluctuates	Reduced if baseline matches seasonal patterns

Trade‑off Discussion

Static thresholds give you absolute control. If your SLA says “response time must stay below 2 s”, a static threshold enforces that rule. It’s straightforward to audit – you can see the exact value in your Terraform repo. However, if your application experiences predictable nightly spikes, a static threshold may generate many false alerts, increasing noise and operator fatigue.

Baseline conditions adapt to your metric’s natural rhythm. New Relic calculates a 95th‑percentile baseline for each hour of the week and compares the live value against it. This reduces noise for expected patterns but can miss a slow, linear degradation that stays within the baseline envelope. Baselines also require at least a month of data before they become reliable, so they’re not suitable for newly deployed services.

Because both types can coexist, a common pattern is to use a baseline for “warning” severity and a static threshold for “critical” severity. That way you get early signals for unusual behavior while still enforcing hard limits.

Concrete Implementation – Terraform Example

Below is a Terraform snippet that creates an alert policy with both a static threshold and a baseline condition for the APM metric apm.transaction.duration (average response time). Replace placeholders with your own values.

# Provider configuration
provider "newrelic" {
  account_id = var.nr_account_id
  api_key    = var.nr_api_key
}

# Alert policy
resource "newrelic_alert_policy" "apm_policy" {
  name = "APM Response Time Policy"
}

# Static threshold condition (critical)
resource "newrelic_alert_condition" "static_rt" {
  policy_id = newrelic_alert_policy.apm_policy.id
  type      = "apm_application"
  name      = "Static Response Time > 2s"
  enabled   = true
  severity  = "critical"

  # Condition details
  apm_application {
    metric_name = "apm.transaction.duration"
    operator    = "above"
    threshold   = 2000
    aggregation_method = "average"
    aggregation_delay   = 60
    evaluation_offset    = 0
    filter {
      key   = "appName"
      value = ["my‑app"]
    }
  }
}

# Baseline condition (warning)
resource "newrelic_alert_condition" "baseline_rt" {
  policy_id = newrelic_alert_policy.apm_policy.id
  type      = "apm_application"
  name      = "Baseline Response Time"
  enabled   = true
  severity  = "warning"

  apm_application {
    metric_name = "apm.transaction.duration"
    aggregation_method = "average"
    aggregation_delay   = 60
    evaluation_offset    = 0
    filter {
      key   = "appName"
      value = ["my‑app"]
    }
    baseline {
      # No parameters needed – New Relic calculates automatically
    }
  }
}

After applying the plan, verify the conditions in the New Relic UI:

1. Navigate to Alerts & AI > Alert conditions.
2. Open the newly created policy and confirm both conditions appear.
3. Use the Preview graph on the baseline condition to ensure at least 4 weeks of data are available.
4. Temporarily lower the static threshold to 1000 ms, trigger a test violation by simulating a slow transaction, and confirm that an incident is created and the notification channel receives the alert.
5. Check the Violation history tab to ensure no false positives during normal variance.

Validation Checklist

• Historical Data – Baseline conditions must show a preview graph; otherwise, the condition will not fire.
• Severity Mapping – Verify that critical alerts trigger escalation policies while warning alerts do not.
• Notification Channels – Test that the configured Slack or PagerDuty channel receives the message.
• Close Rules – Ensure that the policy’s “Close incidents when conditions no longer violated” option is enabled to avoid stale incidents.

By following this decision guide and the example, you can select the appropriate alert type for each metric, balance noise against compliance, and maintain automation through Terraform.