Eject Unhealthy Upstream Hosts in Envoy with Outlier Detection

Problem: a single bad host drags down the whole cluster

When one upstream instance starts returning 5xx or high latency, Envoy keeps sending it traffic unless you intervene. Outlier detection provides passive, per-host ejection based on observed request outcomes. It complements active health checks and is intended to reduce failures and cascading load while allowing automatic reintegration.

Desired outcome

Automatic, gradual ejection of hosts that exhibit error or latency anomalies, with traffic shifted to healthy hosts. Ejected hosts are held out for a base ejection time and then slowly reintroduced. Reintegration happens without manual restarts when the host recovers.

Prerequisites

An Envoy proxy running a version where outlier_detection is supported in the Cluster config. Field names and defaults are version-sensitive, so verify against the version in use.

• Cluster configuration via xDS or static bootstrap for the upstream you want to protect.
• A working request path to the upstream. Outlier detection is passive; it observes real traffic. Active health checking is still recommended for hard failures.
• Admin access to the Envoy admin interface, typically localhost:9901, to inspect stats and config dumps. Requires network access and read permission to admin.

Risk: outlier detection does not replace health checks. Relying on it alone can delay detection of a host that stops accepting connections entirely.

Configure outlier detection for a cluster

Outlier detection lives inside the upstream Cluster. The key knobs are interval, consecutive 5xx threshold, request volume, base ejection time, max ejection percent, and panic threshold.

clusters:
- name: backend
  type: STRICT_DNS
  lb_policy: LEAST_REQUEST
  load_assignment:
    endpoints:
    - lb_endpoints:
      - endpoint:
          address: { socket_address: { address: 10.0.0.10, port_value: 8080 } }
      - endpoint:
          address: { socket_address: { address: 10.0.0.11, port_value: 8080 } }
  outlier_detection:
    consecutive_5xx: 5
    interval: 10s
    base_ejection_time: 30s
    max_ejection_percent: 50
    request_volume: 100
    success_rate_minimum_hosts: 5
    success_rate_request_volume: 100
    success_rate_stdev_factor: 1900
    panic_threshold_percent: 10

What this config intends:

• consecutive_5xx ejects a host after N consecutive 5xx responses observed in the evaluation interval.
• interval is how often Envoy evaluates each host.
• base_ejection_time is the initial hold-out time. It grows with repeated ejections.
• max_ejection_percent caps the fraction of hosts that can be ejected at once to avoid over-ejection.
• panic_threshold_percent disables ejection when the fraction of ejected hosts exceeds this value, protecting against cluster-wide failure.
• success_rate_* enables success-rate based ejection, which compares a host's success rate to the cluster average.

Apply the config via your control plane. With xDS, push a new Cluster resource. With static config, reload Envoy. A full proxy restart is not required for xDS updates.

Verify ejection is working

Check admin stats for outlier ejection counters. Run from a host that can reach the admin port, as the Envoy process owner or with admin network access.

curl -s http://localhost:9901/stats?filter=outlier

Look for counters such as cluster.backend.outlier_detection.ejections and gauges for ejected hosts. Per-host ejection state is visible in the cluster config dump.

curl -s http://localhost:9901/config_dump | jq '.configs[] | select(.@type|contains("Cluster")) | .cluster.outlier_detection'

Expected checks:

• Ejection counters increase when a host crosses the threshold.
• Request success rate and latency distribution improve for the cluster after ejection.
• Ejection timers decay as configured; hosts are reinstated gradually.

Observe upstream metrics from your monitoring system to confirm traffic shifts away from degraded hosts. Do not assume ejection from stats alone.

Recovery and tuning

Outlier detection is self-healing. Once the ejection timer expires, Envoy sends a small probe of traffic to the host. If the host remains healthy, it is fully reinstated.

Recovery options when ejection behaves poorly:

• Increase request_volume and consecutive_5xx to avoid ejection on transient spikes. Aggressive thresholds can eject healthy hosts and reduce capacity.
• Raise base_ejection_time if flapping hosts are reintroduced too quickly.
• Lower max_ejection_percent to limit blast radius during partial degradation.
• Adjust panic_threshold_percent to prevent mass ejection during widespread upstream issues.
• Manually clear ejection via config reload or increasing health check success to trigger reintegration.

Limitations: outlier detection is passive and requires sufficient request volume to evaluate. Low traffic services may not trigger ejection in time.

Practical check after change

After pushing a threshold change, inspect the config dump to confirm the new values are active without a restart. Then monitor ejection counters and upstream error rate for a period covering at least two evaluation intervals. Document the thresholds you chose and the observed ejection rate to guide future tuning.