Diagnosing High CPU and Latency in Envoy Caused by Misconfigured Local Rate Limit Filter

Recognizable condition

Envoy’s admin interface shows elevated CPU usage (cpu.thread_utilization > 80 %) and increased request latency (downstream_rq_time p99 > 200 ms) while overall request throughput stays within expected bounds. The ratelimit.local statistics often display a pattern where tokens_consumed approaches or exceeds tokens_refilled over short intervals, indicating frequent token‑bucket exhaustion.

Cause and diagnostic table

Symptom	Likely cause	Check
High CPU + latency spikes	Rate‑limit token starvation (bucket empties quickly)	Inspect admin /ratelimit/local and /stats for filter.*tokens_consumed vs filter.*tokens_refilled
Frequent 429 responses	Over‑strict limit (max_requests/unit too low)	Review route configuration’s local_rate_limit.action and max_requests/unit values
No change after reload	Stale dynamic configuration (xDS not delivering updated RateLimitService config)	Confirm xDS version and that the management server has pushed the latest RateLimitService proto

Ordered checks

1. Enable debug logging for the local_rate_limit filter via the admin logging endpoint (requires ability to POST to /logging on the admin port, typically localhost:18000):

   curl -X POST "localhost:18000/logging?local_rate_limit=debug"
   

   This yields detailed lock‑contention and token‑bucket messages in Envoy’s log stream.

2. Capture a short trace of /stats every 5 seconds for 2 minutes to observe token dynamics:

   for i in {1..24}; do
     curl -s localhost:18000/stats | grep ratelimit.local
     sleep 5
   done
   

   Look for periods where tokens_consumed spikes close to tokens_refilled or where the bucket reports zero available tokens.

3. If logs show frequent lock_contention messages, note that the filter’s internal mutex is being heavily contested—a sign of token starvation.

4. Compare the observed consumption rate to the configured refill rate. Approximate consumption per second can be derived from the delta of tokens_consumed between samples.

5. When consumption ≈ refill and the bucket empties within a few hundred milliseconds, the burst size (max_tokens) or refill rate (tokens_per_fill) is too low for the traffic pattern.

Fixes tied to findings

• Increase the token bucket’s max_tokens (burst) and/or tokens_per_fill (refill rate) proportionally to the measured traffic. Example configuration snippet for a route:

  local_rate_limit:
    stat_prefix: ingress_limits
    token_bucket:
      max_tokens: 2000   # increased from 500
      tokens_per_fill: 200   # increased from 50
      fill_interval: 1s
  

  Adjust values based on the observed tokens‑per‑second needed to keep tokens_consumed well below tokens_refilled.

• If the workload is high‑throughput and low‑latency, consider moving rate limiting to an external RateLimitService (gRPC) to offload token‑bucket work from Envoy workers. This requires configuring a rate_limit_service block and referencing it via the local_rate_limit filter’s rate_limit_service field.

• After applying changes, reload Envoy (e.g., envoy -c /etc/envoy/envoy.yaml --restart-epoch 1 or via your orchestration platform) and monitor the admin endpoints.

Verification

To confirm the fix:

1. Reload Envoy with the updated configuration.

2. Run a steady load test (e.g., wrk -t4 -c100 -d2m http://localhost:80/) and collect stats:

   curl -s localhost:18000/stats | grep -E 'cpu\.thread_utilization|downstream_rq_time p99|ratelimit\.local'
   

3. Check that cpu.thread_utilization stays below 50 % and downstream_rq_time p99 returns to the pre‑issue baseline (commonly < 100 ms).

4. Verify that over a 30‑second window tokens_consumed remains significantly lower than tokens_refilled (e.g., tokens_consumed < 30 % of tokens_refilled).

Limitations

• The local_rate_limit filter is not intended for high‑throughput, low‑latency paths; for such cases prefer an external RateLimitService or kernel‑level mechanisms like BPF.
• Increasing burst or refill rates without traffic‑shape analysis can lead to either excessive throttling (more 429 responses) or insufficient protection (resource exhaustion). Validate changes in a staging environment first.
• Dynamic configuration updates via xDS must be version‑compatible; mismatched versions can cause the filter to retain stale settings even after a reload.

Escalation criteria

If after adjusting max_tokens and tokens_per_fill the CPU remains above 70 % or the adjustment causes an unacceptable rise in 429 errors, escalate to the traffic‑management team to:

• Review the overall rate‑limiting architecture (consider hierarchical rate limiting or client‑side throttling).
• Evaluate enabling the HTTP connection manager’s global_rate_limit filter as an alternative.
• Assess whether workload characteristics necessitate a dedicated external RateLimitService.

---

Note: All commands assume access to Envoy’s admin interface on localhost:18000 with sufficient privileges to POST to /logging and GET /stats. Replace localhost:18000 with your actual admin address and port as needed.