Diagnosing and Fixing Envoy Outlier Detection Storms

Recognize the Symptom

When a cluster starts ejecting hosts, clients see a sudden spike in 503 Service Unavailable responses. In Envoy logs you’ll see entries such as:

2026-10-10T09:00:00Z info envoy: Ejected host 10.0.0.5 from cluster myservice due to consecutive_5xx

These ejections are intentional: Envoy removes a host from load‑balancing when it thinks the host is unhealthy. If too many hosts are ejected, the cluster may become effectively empty, causing the 503 surge.

Common Root Causes

Outlier detection can trigger for several reasons. The most frequent triggers are:

• Consecutive 5xx errors – a host returns many errors in a row.
• Low success rate – the ratio of successful requests falls below a threshold.
• High request volume with failures – a sudden traffic burst reveals a flaky host.
• Detection interval mis‑set – the interval is too short, causing quick ejections.
• Upstream flapping – the host repeatedly goes up and down, confusing Envoy.

Diagnostic Checklist

Follow this ordered checklist to pinpoint the cause.

1. Confirm the spike
   • Check /stats for outlier_ejected_total and cluster_name.upstream_rq_503.

     curl -s http://localhost:9901/stats?format=json | jq '.values | map(select(.name | contains("outlier_ejected_total")))'
     

   • Verify the 503 rate has increased beyond baseline (e.g., >5% of traffic).
2. Identify ejection reasons
   • Search logs for "Ejected host" and note the reason field.

     grep "Ejected host" /var/log/envoy.log | tail -n 20
     

   • Common reasons: consecutive_5xx, success_rate_threshold, request_volume, detection_interval.
3. Inspect cluster configuration
   • Dump the current config and locate the outlier_detection block.

     curl -s http://localhost:9901/config_dump | jq '.dynamic_config.dump.clusters[] | select(.name=="myservice") | .outlier_detection'
     

   • Take note of consecutive_5xx, success_rate_threshold, request_volume, detection_interval, and min_request_volume.
4. Correlate traffic patterns
   • Compare Envoy metrics (e.g., cluster_name.upstream_rq_total) with upstream service logs to see if a traffic burst coincides with the ejection.

Tuning Strategies

Adjust the outlier detection settings only after you understand the root cause. Below are the most common adjustments and when to apply them.

Root Cause	Diagnostic Indicator	Recommended Fix
Consecutive 5xx errors	Logs show consecutive_5xx ejection reason.	Increase consecutive_5xx threshold or patch the upstream service to reduce errors.
Low success rate	Logs show success_rate_threshold reason; cluster_name.upstream_rq_success < 90%.	Raise success_rate_threshold (e.g., from 90% to 95%) or increase min_request_volume to avoid small‑sample swings.
High request volume with failures	Logs show request_volume reason; cluster_name.upstream_rq_total spikes.	Increase request_volume threshold or extend detection_interval to give the host time to recover.
Detection interval too short	Frequent ejections within seconds.	Set detection_interval to at least 10s or more, depending on traffic patterns.
Upstream flapping	Host status toggles between healthy/unhealthy in logs.	Enable interval_jitter_percent or add a failure_mode to ignore transient failures.

Example: Raising the 5xx Threshold

Assume the cluster myservice is ejecting on consecutive_5xx. The current config is:

outlier_detection:
  consecutive_5xx: 5
  interval: 10s
  base_ejection_time: 30s
  max_ejection_percent: 50

To reduce false positives, change consecutive_5xx to 10:

outlier_detection:
  consecutive_5xx: 10
  interval: 10s
  base_ejection_time: 30s
  max_ejection_percent: 50

Apply the new config via a hot reload or restart. After deployment, verify that outlier_ejected_total stops rising and 503 rates return to baseline.

Verification

After making changes, perform the following checks:

1. Stats trend
   • Run /stats over a 5‑minute window and confirm outlier_ejected_total is flat.
2. Error rate
   • Verify cluster_name.upstream_rq_503 is back to normal (<10% of total).
3. Host availability
   • Check /healthcheck or access logs to ensure previously ejected hosts are receiving traffic again.
4. Latency
   • Monitor request latency; a sudden drop indicates the cluster is healthy.

Escalation Criteria

If after tuning the ejection rate remains high or the cluster is still unable to serve traffic, involve the SRE team. Escalate when:

• More than 5% of requests return 503 for over 5 minutes.
• Ejections continue despite threshold adjustments.
• Upstream service health metrics (e.g., CPU, memory, error logs) indicate degradation.
• Potential cascading failures are observed in downstream services.

During escalation, share:

• Current outlier detection config.
• Recent outlier_ejected_total and upstream_rq_503 metrics.
• Upstream service logs showing error spikes.
• Any recent traffic bursts or deployment events.

Best Practices

• Use runtime overrides only for temporary experiments; persist changes in the static config.
• Set min_request_volume high enough to avoid reacting to a handful of requests.
• Combine outlier detection with circuit_breakers and load_shedding for layered resilience.
• Monitor outlier_ejected_total continuously; set alerts for sudden increases.
• Document any threshold changes and the rationale to aid future debugging.