Choosing the Right Load Balancing Policy in Envoy Proxy

The Distribution Dilemma

When configuring an Envoy cluster, the default load balancing behavior often suffices for simple services. However, as traffic patterns shift—such as introducing long-lived gRPC streams or implementing heavy local caching—a generic distribution strategy can lead to "hot spotting," where a single backend instance is overwhelmed while others remain idle.

The goal is to match the lb_policy to the specific nature of your backend workload to minimize latency and maximize resource utilization.

Comparing Envoy Load Balancing Strategies

Envoy provides several policies within the Cluster configuration. The following table compares the three most common strategies for general service mesh traffic.

Policy	Selection Logic	Primary Benefit	Primary Risk
Round Robin	Sequential rotation	Predictable distribution	Ignores backend load/capacity
Least Request	Fewest active requests	Prevents "noisy neighbor" lag	Potential "herding" on new hosts
Maglev	Consistent Hashing	High cache hit rates	Uneven distribution if keys skew

Trade-offs and Decision Drivers

Round Robin: The Baseline

Use Round Robin when your backend instances are identical in capacity and the requests have a uniform processing cost. It is the safest choice for stateless APIs where request latency is consistent across all calls.

Least Request: Optimizing for Latency

Least Request is critical when backend performance varies. For example, if one instance is running on a slower CPU or is experiencing garbage collection pauses, it will naturally accumulate more active requests. Envoy detects this and routes new traffic to faster instances.

Warning: When a new host joins the cluster or a crashed host recovers, it starts with zero active requests. This can cause a "herding" effect where Envoy floods the fresh host with traffic before it can signal load, potentially crashing it again.

Maglev: Optimizing for State

Maglev (Consistent Hashing) is used when the backend maintains a local cache or session state. By ensuring a specific request (based on a hash key) always hits the same host, you reduce the need for expensive distributed cache lookups.

Unlike simple modulo hashing, Maglev minimizes the number of keys remapped when the cluster scales up or down, preventing a "cache storm" where all backends suddenly miss their caches simultaneously.

Implementation Example: Least Request

To implement the Least Request policy, you must modify the clusters section of your envoy.yaml. This example assumes Envoy v1.20+.

clusters:
  - name: backend_service
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: LEAST_REQUEST
    load_assignment:
      cluster_name: backend_service
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: backend-svc.default.svc.cluster.local
                    port_value: 8080

Deployment and Validation

1. Apply Configuration: Update your Envoy configuration and reload the process or apply the change via the xDS API.
2. Generate Load: Use a tool like fortio or wrk to send a sustained burst of requests to the Envoy listener.
3. Monitor Metrics: Check the Envoy admin interface (usually port 9000) or your Prometheus exporter. Look for the upstream_rq_active metric across your backend hosts.

Expected Result: In a Least Request setup, the upstream_rq_active values across all healthy hosts should remain relatively balanced, even if one backend is artificially slowed down (e.g., via a sleep command in the application code).

Limitations and Rollback

Load balancing policies are configured at the cluster level. You cannot change the policy on a per-request basis using standard route configuration; this would require a complex filter chain to route traffic to different clusters with different policies.

Rollback: To revert a policy change, modify the lb_policy value back to ROUND_ROBIN in the envoy.yaml and restart/reload Envoy. This change is non-destructive to the data plane but will immediately shift how traffic is distributed.