Stopping the Bleed: Using Envoy Outlier Detection for Self-Healing Clusters

The Problem: The 'Zombie' Instance

You have a cluster of ten backend pods. Nine are performing perfectly, but one has a corrupted local cache or a leaking connection pool. It isn't "down" in the sense that the TCP port is closed, and it still passes basic active health checks (like a /healthz endpoint that only checks if the process is running). However, 20% of your users are suddenly seeing 503 errors or experiencing 10-second timeouts because their requests happened to hit that one bad instance.

In a standard load-balancing setup, this "zombie" instance continues to receive traffic because it is technically "healthy." To solve this, you need a mechanism that reacts to actual request failures in real-time rather than relying on a periodic heartbeat. This is where Envoy's Outlier Detection comes in.

How Outlier Detection Works

Outlier detection is a passive health check. While active health checks ask a host "Are you okay?", outlier detection watches the actual traffic and asks "Are you behaving?" If a host exceeds a defined threshold of failures, Envoy ejects it from the load-balancing pool for a set period.

Envoy tracks several types of failures to trigger ejection:

• Consecutive 5xx: Ejects the host after a specific number of consecutive 5xx errors.
• Success Rate: Ejects the host if the ratio of successful requests falls below a percentage.
• Consecutive Gateway Failures: Ejects the host if it fails to connect or returns gateway errors.

Crucially, ejection is temporary. After the base_ejection_time expires, Envoy allows the host back into the rotation. If the host fails again immediately, Envoy typically increases the ejection duration, creating an exponential backoff that prevents a flapping service from destabilizing the cluster.

Configuration Example: The "Fast-Fail" Pattern

The following configuration is designed for a service where a few consecutive errors are a strong signal of a failing instance. This example assumes Envoy v1.20+.

clusters:
- name: backend_service
  connect_timeout: 0.25s
  type: STRICT_DNS
  lb_policy: ROUND_ROBIN
  load_assignment:
    cluster_name: backend_service
    endpoints:
    - lb_endpoints:
      - endpoint:
          address: 
            socket_address: { address: backend-svc, port_value: 8080 }
  common_http_protocol_options: {}
  outlier_detection:
    # Check statistics every 10 seconds
    interval: 10s
    # Keep the host out for 30 seconds initially
    base_ejection_time: 30s
    # Eject after 2 consecutive 5xx errors
    consecutive_5xx: 2
    # Never eject more than 50% of the cluster
    max_ejection_percent: 50
    # If more than 50% are unhealthy, ignore health and send traffic to all
    healthy_panic_threshold: 50

What happens here?

1. If a pod returns two 500 Internal Server Error responses in a row, Envoy marks it as an outlier.
2. The pod is removed from the load balancing rotation for 30 seconds.
3. If 6 out of 10 pods fail, the max_ejection_percent (50%) prevents the 6th pod from being ejected.
4. If the failure rate spikes so high that the number of healthy hosts drops below the healthy_panic_threshold, Envoy enters "panic mode." In this state, it ignores all ejection logic and sends traffic to every host, assuming that some response is better than no response.

Operational Trade-offs

Outlier detection is powerful, but it introduces specific risks that you must manage:

Local vs. Global State

Ejection decisions are local to each Envoy process. If you have 100 sidecars, each sidecar maintains its own statistics. Sidecar A might eject Pod 1, while Sidecar B still thinks Pod 1 is healthy. This is generally acceptable, but it means your observability must aggregate these ejections across the fleet to see the full picture.

The Thundering Herd

When a large group of hosts is ejected and then returns simultaneously after the base_ejection_time, they may be slammed with a surge of traffic. If the original cause of the failure was load-related, this can trigger a secondary crash. Pair outlier detection with circuit breaking (max connections/requests) to limit the impact of returning hosts.

Verifying Ejections

To confirm that outlier detection is working, you can query the Envoy admin interface (typically on port 9901). Use the following checks:

• Stats Check: Run curl http://localhost:9901/stats | grep outlier. Look for outlier_detection.ejections_active to see how many hosts are currently sidelined.
• Cluster Status: Run curl http://localhost:9901/clusters. Look for the specific host endpoints; an ejected host will be flagged with FAILED_OUTLIER_CHECK.

Summary Checklist

Before deploying outlier detection to production, verify the following:

• [ ] Validate Config: Run envoy --mode validate -c your-config.yaml to ensure field names match your Envoy version.
• [ ] Set Panic Threshold: Ensure healthy_panic_threshold is set to avoid total cluster blackout during a widespread outage.
• [ ] Monitor Ejections: Create an alert on the ejections_active stat so you know when your cluster is self-healing (or failing).