Using Envoy Outlier Detection to Automatically Eject Unhealthy Upstreams

Problem: flaky upstream causing latency spikes

When one instance in a service cluster starts returning errors or slowing down, Envoy keeps sending it traffic unless you intervene manually. The result is increased latency, failed requests, and wasted capacity while the unhealthy host recovers.

Thesis: Envoy’s outlier detection can automatically eject bad hosts and reduce traffic churn

Outlier detection continuously measures per‑host metrics such as success rate, request volume, and latency. When a host crosses a configurable threshold, Envoy stops sending new requests for an ejection period that grows with each repeat offense, giving the instance time to recover while protecting the rest of the mesh.

How outlier detection works

For each upstream host Envoy tracks:

• Number of consecutive 5xx responses (or connection failures for TCP)
• Success rate over a sliding window
• Average request latency

If any metric exceeds its configured limit, the host is marked an outlier. Envoy then ejects it for base_ejection_time multiplied by 2^(n‑1) where n is the ejection count for that host. The ejection period caps at max_ejection_percent of the cluster’s hosts.

Configuration example

Below is a minimal static cluster with two upstreams: one healthy (returns 200) and one unhealthy (returns 500). The snippet shows how to enable outlier detection with aggressive but understandable settings.

static_resources:
  clusters:
  - name: test_cluster
    connect_timeout: 0.25s
    type: STATIC
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: test_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: 127.0.0.1, port_value: 8080 }
        - endpoint:
            address:
              socket_address: { address: 127.0.0.1, port_value: 8081 }
    outlier_detection:
      consecutive_5xx: 1          # eject after a single 5xx
      interval: 10s               # evaluation interval
      base_ejection_time: 30s     # first ejection lasts 30s
      max_ejection_percent: 50    # at most half the cluster can be ejected
      enforcing_consecutive_5xx: true
      success_rate_minimum_hosts: 1   # require at least one host for success rate calc
      success_rate_request_volume: 10
      success_rate_stdev_factor: 1900  # essentially disable success‑rate check

Place this snippet in your Envoy bootstrap or dynamic cluster configuration and reload Envoy.

Trade‑offs and limitations

Setting consecutive_5xx to 1 and a low base_ejection_time can cause frequent ejections, especially in small clusters, leading to traffic churn and latency spikes. Moreover, outlier detection relies on correct classification of success versus failure; counting 4xx client errors as successes will hide real problems or trigger false positives. Operators should align the success criteria with their application’s semantics and start with conservative values, then observe the impact.

Practical verification steps

1. Start Envoy with the configuration above.
2. Send a steady stream of requests (e.g., using hey or wrk) to the cluster.
3. Check the admin interface: curl localhost:9901/stats (adjust port if needed). Look for rising values in outlier_detection.ejection_count and a drop in upstream_rq_200 for the unhealthy host.
4. Confirm that no new requests are sent to the unhealthy host during the ejection period by examining Envoy access logs or tracing.
5. After base_ejection_time seconds, verify that traffic resumes to that host and the ejection count stops increasing.

These steps let you observe the back‑off behavior without claiming any specific test outcome.

Actionable closing

Begin with modest thresholds (e.g., consecutive_5xx: 3, base_ejection_time: 15s) and monitor the ejection statistics. Adjust only after you see a clear pattern of unhealthy hosts causing latency or error spikes. Pair outlier detection with retries, timeouts, and circuit breaking for layered resilience, and regularly review the success‑rate criteria to ensure they match your application’s definition of a successful request.