Handling Gray Failures with Envoy Outlier Detection

In a microservice architecture, a single degraded instance can cripple an entire request chain. When one pod in a cluster starts returning intermittent 5xx errors or timing out, standard load-balancing algorithms often continue to send it traffic. This leads to a 'retry storm' where clients repeatedly hit the failing node, increasing tail latency and potentially cascading failures across your system.

The most effective way to handle these 'gray failures'—where a node is technically alive but functionally broken—is through Envoy outlier detection. Instead of waiting for a manual intervention or a health check to fail, Envoy can temporarily eject flaky hosts from the load balancing pool based on real-time request performance.

Passive Health Checking vs. Active Probing

Unlike active health checks, which ping a specific endpoint (like /health) at set intervals, outlier detection is passive. It monitors the actual results of live traffic flowing through the proxy. If an upstream host exceeds a predefined threshold of failures, Envoy marks it as an 'outlier' and removes it from the load balancing set for a cool-down period.

Envoy tracks several metrics to make these decisions:

• Consecutive Gateway Errors: Ejection occurs if a host returns a specific number of 5xx errors in a row.
• Success Rate: Ejection occurs if the percentage of successful requests falls below a threshold over a sliding window.
• Local Concurrency: Ejection occurs if one host has significantly more active requests than its peers, suggesting it is hanging.

Configuration Example: Protecting a Service Cluster

To implement this, you define the outlier_detection block within your cluster configuration. This example assumes Envoy version 1.18 or newer. We will configure a cluster to eject hosts that produce five consecutive 5xx errors.

clusters:
  - name: backend_service
    type: STRICT_DNS
    load_balancing_policy: ROUND_ROBIN
    outlier_detection:
      consecutive_5xx: 5
      base_ejection_time: 30s
      max_ejection_time: 5m
      max_ejection_percentage: 10

The parameters function as follows:

• consecutive_5xx (5): The host is ejected if it returns 5 server errors back-to-back.
• base_ejection_time (30s): The initial duration the host is kept out of the pool.
• max_ejection_time (5m): The absolute maximum time a host can be ejected, even if it continues to fail upon re-entry.
• max_ejection_percentage (10%): A safety circuit. Envoy will never eject more than 10% of the cluster. This prevents a total outage if the entire service is failing globally.

Verification and Monitoring

To verify that Envoy is identifying and ejecting outliers, query the Envoy admin interface (typically port 9001) from the local host. You will need permissions to access the admin port.

# Check for active ejections in the backend_service cluster
curl http://localhost:9001/stats | grep cluster.backend_service.outlier_detection

Look for the outlier_detection.ejections_active metric. If this value is greater than zero, Envoy has successfully removed flaky nodes from the load-balancing pool. You can also monitor upstream_rq_consecutive_5xx to see how close hosts are to the threshold.

The Trade-off: Aggression vs. Stability

The primary risk with outlier detection is over-ejection. If the consecutive_5xx threshold is set too low (e.g., 1 or 2), a momentary network blip could cause Envoy to remove healthy nodes. This increases the load on the remaining nodes, which may then fail under the added pressure, creating a death spiral.

To mitigate this, always set a conservative max_ejection_percentage and ensure your base_ejection_time allows the upstream service enough time to recover or be replaced by an orchestrator like Kubernetes.

Rollback Procedure

If you observe an increase in 503 errors due to over-ejection, remove the outlier_detection block from the cluster configuration and reload the Envoy configuration via the /reload` endpoint or by restarting the process.