Creating a Datadog SLO for Latency and Configuring Burn‑Rate Alerts

Desired Outcome

Create a Service Level Objective (SLO) that measures the percentage of requests with latency under 200 ms over a rolling 7‑day window, and configure burn‑rate alerts that notify the team when the error budget is being consumed too quickly.

Prerequisites

• A Datadog organization with access to the Monitors, SLOs, and Alerts modules.
• User role that allows creating monitors, SLOs, and alert policies (typically Admin or a custom role with slo_write and monitor_write permissions).
• An existing monitor or metric query that captures request latency (e.g., a Datadog APM latency monitor or a custom metric from http.server.latency).

Procedure

1. Identify or create the latency query

   If you already have a monitor that alerts on latency >200 ms, note its query ID. Otherwise, create a monitor:

   # Example: create a monitor via the Datadog API (run in a terminal with API & APP keys)
   curl -X POST "https://api.datadoghq.com/api/v1/monitor" \
     -H "Content-Type: application/json" \
     -H "DD-API-KEY: " \
     -H "DD-APPLICATION-KEY: " \
     -d '{
       "type": "metric alert",
       "query": "avg(last_5m):avg:http.server.latency{*} > 200",
       "name": "Latency >200ms (5‑min avg)",
       "message": "Latency exceeded 200 ms on {{host.name}}.",
       "tags": ["service:frontend"],
       "options": {
         "thresholds": {
           "critical": 200
         }
       }
     }'
   

   Save the monitor ID returned in the response (e.g., 123456).

2. Create the SLO from the monitor
   1. In the Datadog UI, navigate to Monitors → Manage Monitors, locate the monitor you just created, and click its name.
   2. On the monitor detail page, click Create SLO (or go to Service Level Objectives → New SLO and select Monitor-based).
   3. Select the monitor (by ID or name) as the source.
   4. Set the Target to 99.9% (meaning 99.9% of evaluations must be ok, i.e., latency ≤200 ms).
   5. Choose a Time window of 7d (rolling).
   6. Provide a clear name, e.g., frontend_latency_slo, and optionally add tags like team:frontend.
   7. Click Create SLO.
3. Configure burn‑rate alerts
   1. From the SLO list (Service Level Objectives), open the newly created SLO.
   2. In the SLO detail view, click Create Burn‑rate Alert.
   3. Choose an alert condition; a common starting point is:
   • Burn rate > 1.0 over 1h (i.e., consuming the error budget faster than expected).
   4. Set the notification channels (e.g., Slack, email, PagerDuty).
   5. Provide a descriptive alert name, such as frontend_latency_slo_burnrate_1h.
   6. Save the alert.
4. Link the SLO to a dashboard (optional but recommended)
   1. Open or create a dashboard where you want to display reliability metrics.
   2. Add a SLO widget and select the SLO you just created.
   3. Optionally add a Burn‑rate widget to visualize consumption over time.
   4. Save the dashboard.

Expected Checks

• After creation, the SLO appears in the Service Level Objectives list with a status of OK or Warning based on recent data.
• The SLO detail page shows:
• Current compliance percentage (should be close to 100 % if latency is good).
• Error budget remaining (e.g., 0.1 % of the 7‑day window).
• Burn‑rate graph with the alert threshold visible.
• If you generate latency spikes (e.g., by injecting delay in a test service), the burn‑rate should rise and the configured alert should fire after the defined evaluation period (e.g., 1 hour). Verify receipt in your notification channel.
• The dashboard widgets update in near‑real time reflecting the SLO and burn‑rate values.

Recovery Options

If the SLO or its underlying monitor needs to be changed:

1. Edit the monitor: Changing the monitor query does not> automatically update the SLO. After editing, open the SLO and click Edit to reselect the monitor or adjust the query.
2. Adjust the SLO target or window: Use the Edit SLO button to modify the target percentage or time window; note that this will recalculate the error budget from the start of the new window.
3. Delete the SLO: If the SLO is no longer needed, select it in the list and choose Delete. This removes associated burn‑rate alerts unless they are manually retained.
4. Rollback alert changes: Burn‑rate alerts can be edited or deleted from the Monitors → Manage Monitors view (they appear as regular metric alerts).

Always verify after any change that the SLO still reflects the intended latency target and that alerts fire as expected.