Datadog SLOs That Track What Users Actually Feel

The alert was green while users were seeing errors

Your infrastructure dashboards show CPU, memory, and error rates all within normal limits, but support tickets are climbing. The gap usually comes from measuring the wrong thing: host health is not the same as user success. A service can return 200 OK while returning empty shopping carts, or a database can be up while checkout times out.

The useful takeaway is to define a Datadog service level objective (SLO) around a user-facing service level indicator (SLI), then alert on how fast the error budget is being spent rather than on every individual blip. An SLO is a reliability target, such as 99.9 percent of requests succeeding over 30 days. The error budget is the remaining allowance for failures before that target is missed.

Choose an SLI that matches a user failure

Datadog lets you build an SLI from metrics, monitors, logs, or real user monitoring (RUM). For a web service, a practical starting point is the ratio of successful responses to total responses. Avoid using only infrastructure metrics like CPU utilization, because those do not capture whether a request actually completed.

In the Datadog UI, go to Manage SLOs, create a metric-based SLO, and define a query such as successful requests divided by all requests. A common metric SLI looks like this:

sum:http.requests{status:2xx,env:prod} / sum:http.requests{env:prod} * 100

Set the target to 99.9 percent over a rolling 30-day window. Datadog calculates compliance and shows the error budget remaining in the SLO details page.

Worked example: attach a burn-rate monitor

Compliance alone tells you how you are doing; burn-rate alerts tell you when to act. A burn rate measures how quickly the error budget is being consumed over a short window, such as one hour or six hours, compared with the longer SLO window.

Example decision:

• SLO target: 99.9 percent successful requests over 30 days.
• Fast burn alert: more than 2 percent of the 30-day budget consumed in one hour.
• Slow burn alert: more than 5 percent of the budget consumed in six hours.

If a deployment causes a spike in 5xx errors, the fast burn monitor can page on-call before the 30-day compliance drops visibly. If the error rate creeps up gradually, the slow burn monitor catches chronic regression.

To create this through the Datadog API, you would typically authenticate with an API key and application key, then submit the SLO definition to the SLO endpoint. Placeholders below show the shape of the call; verify exact field names and rate limits against your current Datadog plan and API documentation before automating.

curl -X POST "https://api.datadoghq.com/api/v1/slo" \
  -H "DD-API-KEY: " \
  -H "DD-APPLICATION-KEY: " \
  -H "Content-Type: application/json" \
  -d '{
    "name": "",
    "description": "User-facing request success rate",
    "type": "metric",
    "target": 99.9,
    "window": 30,
    "metric": {
      "numerator": "sum:http.requests{status:2xx,env:prod}",
      "denominator": "sum:http.requests{env:prod}"
    }
  }'

Required permissions: a Datadog API key with write access to SLOs. Expected check: after creation, the SLO appears in Manage SLOs with the correct target, type, and windows. Risk: changing the time window retroactively changes the calculated budget history, so do not alter windows on an active SLO without understanding the impact.

Limitations and how to verify

SLOs measure compliance, not root cause. A green SLO does not mean your system is healthy; it means the chosen SLI stayed within target. If the SLI misses a user-facing failure mode, the SLO can look healthy while users experience errors. Burn-rate alerts also depend on the query and rollup; a misconfigured denominator can hide failures.

Practical verification:

1. Open Manage SLOs in Datadog and confirm the compliance percentage and error budget appear for your new SLO.
2. Use the Datadog API to list SLOs and compare the returned target, type, and windows against the UI configuration.
3. Optionally, create a test monitor that intentionally violates the SLI and confirm that SLO compliance drops and any attached burn-rate monitor triggers.

Do not run the intentional violation test on production traffic unless you can isolate it to a test service or synthetic traffic.

Closing

Start with one user-facing SLO per critical service, attach burn-rate alerts for fast and slow budget consumption, and review the SLI quarterly to make sure it still matches what users actually feel. SLOs are a compass for reliability, not a substitute for tracing, logs, and actionable alerts.