Choosing the Right Envoy Load Balancing Policy for Upstream Clusters

The Traffic Distribution Dilemma

When configuring an Envoy upstream cluster, the default load balancing behavior may not align with your backend's performance profile. Choosing the wrong policy often manifests as "tail latency spikes," where a few requests take significantly longer than others because they were routed to a host already struggling with a heavy payload, despite other hosts being idle.

The goal is to match the Load Balancing (LB) Policy to the nature of your requests (homogeneous vs. variable) and your infrastructure (identical vs. mixed capacity).

Comparing Envoy LB Policies

Envoy provides several policies to distribute traffic. The following table compares the most common options based on operational constraints.

Policy	Best Use Case	Key Trade-off	State Requirement
ROUND_ROBIN	Identical hardware; uniform request cost.	Ignores backend load/latency.	Stateless
LEAST_REQUEST	Variable request duration (I/O bound).	Slight memory overhead per host.	Active request tracking
MAGLEV / RING_HASH	Caching layers; session persistence.	Risk of "hotspots" with skewed keys.	Consistent Hashing
RANDOM	Very large clusters; low-overhead needs.	Potential for uneven distribution.	Stateless

Decision Logic: When to Switch

From Round Robin to Least Request

Use LEAST_REQUEST when your backend processing time is unpredictable. In a ROUND_ROBIN setup, if Host A receives three "heavy" requests (e.g., complex database queries) and Host B receives three "light" requests (e.g., health checks), Envoy will continue sending the next request to Host A regardless of its mounting queue. LEAST_REQUEST prevents this by routing to the host with the fewest active concurrent requests.

From Least Request to Consistent Hashing

If your backend relies on local in-memory caches to perform efficiently, LEAST_REQUEST is counter-productive because it ignores data locality. MAGLEV or RING_HASH ensures that requests with the same hash key (such as a User ID or Session ID) always hit the same upstream host, maximizing cache hit rates at the cost of potentially uneven load distribution.

Implementation Example: Configuring Least Request

To implement the LEAST_REQUEST policy, you must modify the lb_policy field within the cluster configuration of your envoy.yaml. This example assumes Envoy v1.20+.

clusters:
  - name: backend_service
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: LEAST_REQUEST
    load_assignment:
      cluster_name: backend_service
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: backend-svc.internal
                    port_value: 8080

Validation and Diagnostics

To verify that the policy is functioning, you cannot rely on simple request counts; you must observe the distribution relative to host latency. Use the Envoy Admin Interface (typically on port 9000) to inspect the cluster state.

1. Check Cluster Stats: Run curl http://localhost:9000/stats | grep "cluster.backend_service". Look for upstream_rq_active to see the current number of active requests per host.
2. Verify Distribution: If you simulate a "slow" host (e.g., by introducing a sleep command in one backend instance), LEAST_REQUEST should show a decreasing trend of new requests sent to that host compared to healthy, fast hosts.

Operational Limitations

• Memory Overhead: LEAST_REQUEST requires Envoy to maintain a counter for every active request per host. In environments with tens of thousands of upstream endpoints, this can increase the memory footprint of the Envoy process.
• Health Check Dependency: Load balancing policies only operate on "healthy" hosts. If you have not configured health_checks, Envoy may continue sending traffic to a failing host because it is technically the one with the "least requests" (as it is dropping them immediately).

Rollback Procedure

If you observe increased memory usage or unexpected routing behavior after changing the policy, revert the lb_policy to ROUND_ROBIN in the envoy.yaml and reload the configuration via the admin API or a process restart.