Preventing Service Meltdown with Envoy Local Rate Limiting

The Danger of the Traffic Spike

When a backend service experiences a sudden surge in traffic—whether from a marketing campaign, a retry storm, or a malfunctioning client—the result is often a cascading failure. As the service slows down, requests queue up, memory usage spikes, and eventually, the service crashes. If you have multiple instances of a service, the load shifts to the remaining healthy nodes, knocking them over in a domino effect.

The immediate goal is to protect the backend from total collapse. While global rate limiting (using a centralized store like Redis) provides a strict ceiling on traffic, it introduces network latency and a single point of failure. For many engineering teams, local rate limiting is the more practical first line of defense. It allows each Envoy instance to protect its own upstream targets without needing to talk to a remote coordinator.

How Local Rate Limiting Works

Envoy implements local rate limiting using a token bucket algorithm. Imagine a bucket that refills with "tokens" at a fixed rate per second. Every incoming request that matches a specific rule must "spend" a token to pass through. If the bucket is empty, Envoy immediately rejects the request with an HTTP 429 Too Many Requests response.

Because this happens entirely within the Envoy process, there is virtually no performance penalty. The proxy doesn't wait for a network round-trip to check a quota; it simply checks a local counter in memory.

Implementing a Local Limit

To enable this, you must add the envoy.filters.http.local_rate_limit filter to your HTTP connection manager. You can define limits based on descriptors—which are essentially keys derived from the request (like a header or a path)—or apply a generic limit to all traffic.

Below is a configuration example for an Envoy bootstrap or listener setup. In this scenario, we are limiting requests to a specific /api/heavy-task endpoint to 10 requests per second per Envoy instance, allowing for a small burst of 20.

{
  "name": "envoy.filters.http.local_rate_limit",
  "typed_config":
  {
    "stat_prefix": "local_rate_limiter",
    "token_bucket":
    {
      "max_tokens": 20,
      "tokens_per_fill": 10,
      "fill_interval": { "value": 1, "scale_units": "s" }
    },
    "filter_enabled": "true",
    "filter_stopped_at_local_rate_limit": "true"
  }
}

Calculating the "Global" Impact

The most critical trade-off with local rate limiting is the lack of a shared state. If you configure a limit of 10 requests per second (rps) and you have 5 Envoy proxies deployed in your cluster, your backend is actually exposed to a total of 50 rps.

Metric	Local Rate Limit	Global Rate Limit
Latency	Near Zero	Network Round-trip to Store
Precision	Approximate (Sum of nodes)	Exact (Single counter)
Reliability	High (No external dependency)	Dependent on State Store (e.g. Redis)

Verification and Testing

To verify that your limits are working, you should use a load-testing tool like wrk or hey from a terminal with access to the proxy. Run a test that intentionally exceeds your tokens_per_fill value:

# Example using 'hey' to send 100 requests at a high rate
hey -n 100 -c 10 http://your-envoy-proxy/api/heavy-task

Expected Result: You should see a distribution of 200 OK responses followed by a significant number of 429 Too Many Requests responses once the initial burst (max_tokens) is exhausted.

Practical Limitations

Local rate limiting is a "blunt instrument." Because it is distributed, it cannot prevent a single user from overwhelming the system if that user's traffic is perfectly balanced across all your Envoy instances. It is designed to protect the infrastructure, not to enforce business quotas (like "100 requests per month per user"). For business-level quotas, a global rate limiter is required.

Closing Action

If you are seeing intermittent backend timeouts during traffic spikes, don't start with a complex global coordinator. Implement a conservative local rate limit on your most expensive endpoints. This ensures that while some users may receive a 429, the service remains available for everyone else, preventing a total system blackout.