Envoy Global Rate Limiting: Architecture and Practical Trade-offs

The problem: quotas across a fleet

When you run dozens of Envoy sidecars or edge proxies, per-instance rate limiting isn't enough. A single client can hammer different instances and bypass local counters. You need a single source of truth for quotas — enter the Rate Limit Service (RLS), a separate gRPC service that Envoy calls on every request.

How the descriptor model works

Envoy doesn't send "user 123, path /api" as a raw string. It builds a descriptor — a list of (key, value) tuples — that the RLS evaluates against its rule set. A typical descriptor might look like:

descriptors:
  - entries:
      - key: "remote_address"
        value: "10.0.0.5"
      - key: "request_path"
        value: "/api/v1/payments"
      - key: "generic_key"
        value: "user_123"

The RLS (often the reference implementation in envoyproxy/rate-limit-service) matches these entries against a YAML/JSON config that defines limits like "100 requests per minute per user on /payments". Because the descriptor is arbitrary, you can rate-limit by API key, Kubernetes namespace, or any header you propagate.

Worked example: wiring Envoy to the RLS

Below is a minimal envoy.yaml snippet that enables the HTTP rate limit filter and points it at a local RLS instance. Run this on a test box; the RLS container can be started with docker run -p 8081:8081 -p 8082:8082 envoyproxy/rate-limit-service:latest.

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 10000 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route: { cluster: service_cluster }
          http_filters:
          - name: envoy.filters.http.ratelimit
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
              domain: envoy
              failure_mode_deny: false   # fail-open
              rate_limit_service:
                grpc_service:
                  envoy_grpc:
                    cluster_name: rate_limit_cluster
                transport_api_version: V3
          - name: envoy.filters.http.router
  clusters:
  - name: service_cluster
    type: STATIC
    load_assignment:
      cluster_name: service_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: host.docker.internal, port_value: 8080 }
  - name: rate_limit_cluster
    type: STATIC
    load_assignment:
      cluster_name: rate_limit_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: host.docker.internal, port_value: 8081 }

With this config, every inbound request triggers an async gRPC call to rate_limit_cluster. If the RLS replies OVER_LIMIT, Envoy returns 429 Too Many Requests with a Retry-After header. The failure_mode_deny: false setting means a downed RLS results in allowing the request (fail-open); flip to true for fail-closed.

Latency, failure modes, and the thundering herd

The mandatory gRPC round-trip adds 1–5 ms per request on a local network — acceptable for most APIs, but measurable on hot paths. Two operational knobs matter:

• Timeouts: Set timeout on the grpc_service (default 20 ms). Too short and you get false 429s; too long and you add tail latency.
• Fail-open vs fail-closed: failure_mode_deny chooses the behavior when the RLS is unreachable. Fail-open preserves availability but lets bursts through; fail-closed protects quotas but can turn an RLS outage into a full outage.

Under heavy load, the RLS itself can become a bottleneck. The reference service uses Redis for counter storage; if Redis saturates, every Envoy instance piles onto the same failing backend — the classic thundering herd. Mitigations include:

• Running multiple RLS replicas behind a load balancer.
• Enabling local rate limiting (envoy.filters.http.local_ratelimit) as a first line of defense for known abusive patterns.
• Configuring rate_limit_service.rate_limit_headers to propagate X-RateLimit-Limit and X-RateLimit-Remaining so clients can back off early.

Verify it works

1. Start the RLS and Envoy with the config above.
2. Send a burst: hey -n 200 -c 10 http://localhost:10000/api/v1/payments.
3. Confirm responses shift from 200 to 429 once the descriptor limit is hit.
4. Check Envoy stats: curl localhost:9901/stats | grep ratelimit — look for ratelimit.over_limit increments.

If you see 429s and the counters climb, the integration is functional. Next step: tune the RLS rule config (usually config.yaml mounted into the RLS container) to match your actual quota policy.