Decision Guide: Enabling Outlier Detection in Envoy Proxy Clusters

Problem and Decision

You notice that a subset of upstream hosts occasionally returns 5xx errors, causing increased latency and failed requests for clients. The goal is to let Envoy automatically eject misbehaving hosts for a configurable period, reducing the impact of faulty instances while keeping the system stable.

Constraints

• Envoy version ≥1.18 (outlier detection API stable).
• Ability to reload Envoy configuration via hot‑restart or the admin interface.
• Cluster must have sufficient request volume for statistical thresholds to be meaningful.
• Memory overhead per host is acceptable (≈ a few hundred bytes).

Supported Options

Outlier detection can be driven by two primary modes:

Mode	Key Settings	Typical Use‑Case
Consecutive errors mode	consecutive_5xx, interval, base_ejection_time, max_ejection_percent	Detects spiky bursts of 5xx responses; low CPU, simple to tune.
Success rate mode	success_rate, success_rate_minimum_hosts, success_rate_request_volume, interval, base_ejection_time, max_ejection_percent	Better for gradual degradation; requires enough requests to compute a reliable rate.

Trade‑offs

Consecutive Errors Mode

• Pros: Triggers instantly after N bad responses; minimal computation; works well when errors appear in bursts.
• Cons: May eject hosts during occasional error spikes that are not indicative of a lasting problem (thrashing).

Success Rate Mode

• Pros: Smooths out noise; ejects only when the observed success rate falls below a threshold over a window.
• Cons: Needs a minimum request volume; reacts slower to sudden bursts; slightly higher CPU for rate calculation.

Concrete Implementation

Add an outlier_detection block inside the cluster definition. The example below uses the consecutive‑errors mode, which is often the safest starting point.

clusters:
  - name: backend_service
    connect_timeout: 0.25s
    type: LOGICAL_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: backend_service
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: 10.0.0.1
                    port_value: 8080
    outlier_detection:
      consecutive_5xx: 3          # eject after 3 consecutive 5xx
      interval: 10s               # statistical window
      base_ejection_time: 30s     # initial ejection duration
      max_ejection_percent: 10    # eject at most 10% of hosts

After saving the updated configuration, reload Envoy without dropping connections:

• Hot‑restart: envoy -c /etc/envoy/envoy.yaml --restart-epoch 1 (or use your orchestration tool’s hot‑restart command).
• Admin interface: POST /quitquitquit followed by a new start, or POST /runtime_modify?key=admin.enable_hot_restart&value=true if already enabled.

Validation Steps

1. Generate test traffic that returns a 5xx every third request (e.g., using a simple while true; do curl -s -o /dev/null -w "%{http_code}\\n" http://localhost:10000/test; sleep 0.1; done loop against a test endpoint).
2. Watch the admin stats endpoint: curl -s http://localhost:9901/stats | grep outlier_detection. Look for cluster.backend_service.outlier_detection.ejected_total increasing.
3. Check the health‑check endpoint: curl -s http://localhost:9901/healthcheck/cluster/backend_service. Ejected hosts appear with the flag ea (ejected active).
4. Confirm that the load‑balancing pool temporarily excludes the offending host (e.g., by observing request distribution with envoy --mode read-only --admin-address localhost:9901 or via your service mesh telemetry).

Limitations and Practical Checks

• Each host tracked by outlier detection consumes extra memory; very large clusters (>10k hosts) may see measurable overhead. Monitor cluster..outlier_detection.memory_used via /stats if concerned.
• Aggressive thresholds (low consecutive_5xx or high success_rate) can cause thrashing under bursty traffic. Start with conservative values, then adjust based on observed ejected_total and latency metrics.
• Ensure the cluster receives enough requests per interval for the success‑rate mode to be statistically sound; otherwise the mode may never trigger.

Rollback (if needed)

Since enabling outlier detection changes the running configuration, you can revert by removing the outlier_detection block from the cluster definition and performing another hot‑restart or admin‑initiated reload. Verify that ejected_total stops increasing and that all hosts return to healthy status in the health‑check output.

Summary

Choose consecutive‑errors mode for quick reaction to bursty 5xx spikes; opt for success‑rate mode when you have steady traffic and need smoother, volume‑based detection. Configure the block as shown, reload Envoy, and validate via admin stats and health‑check endpoints. Keep an eye on memory usage and adjust thresholds to avoid unnecessary thrashing.