limit_req burst smoothing or limit_conn hard cap for protecting an upstream during traffic spikes

Primary mechanism

Use limit_req with a rate matching the upstream's sustainable request rate (≈50 r/s) and a modest burst (e.g., 10) as the first line of defense.

Why limit_req first

It smooths short spikes by queuing excess requests, adding latency but avoiding immediate 503s. For interactive clients, a small burst keeps queuing delay within acceptable limits.

Role of limit_conn

Add a limit_conn zone with a concurrency limit equal to the safe concurrent request count (≈50) to provide a hard ceiling that protects against bursts that exceed the queue’s drain rate or against long‑running requests that would otherwise let the average rate creep up.

Combining directives

The two are complementary: limit_req controls average rate and burst size; limit_conn guards the concurrent‑connection budget that limit_req cannot guarantee because request processing time varies.

Setting burst size relative to concurrency budget

Let C be the upstream’s comfortable concurrent heavy requests (≈50). Choose a request rate R the upstream can sustain (e.g., 50 r/s). Set burst B so that the sum of active requests plus queued requests stays ≤ C. A practical rule: set burst to 10‑20 % of the rate (e.g., B = 0.1 × R to 0.2 × R). With R=50 r/s, B=5‑10 gives a maximum of ~10 queued requests; if each request takes ~0.2 s (5 r/s per connection), the queued load stays under C. Adjust based on observed latency.

Version‑specific behavior with proxy buffering

In nginx 1.24/1.25:

• limit_req counts requests as they arrive, before they are passed to proxy_pass. Changing proxy_buffering on/off does not affect this counting.
• limit_conn counts the underlying TCP connections (or $binary_remote_addr keyed connections). It is also independent of proxy buffering.
• If HTTP/2 is enabled, multiple streams share a single connection, so a connection‑based limit_conn may undercount concurrent work. Verify by testing with an HTTP/2 client or consider keying limit_conn on $binary_remote_addr plus $server_port and using limit_req for per‑request rate control.

Verification steps (no changes needed unless HTTP/2)

1. Configure a test location:

   limit_req_zone $binary_remote_addr zone=one:10m rate=50r/s;
   limit_conn_zone $binary_remote_addr zone=addr:10m;
   location /heavy {
       limit_req zone=one burst=10 nodelay;
       limit_conn zone=addr 50;
       proxy_pass http://upstream;
   }
   

2. Run a load generator (wrk, hey) simulating bursts of heavy requests.
3. Observe responses: 200 with added latency indicates queuing; 503 indicates rejection.
4. Check upstream concurrency via stub_status or upstream metrics; ensure it stays ≤ 50.
5. Repeat with proxy_buffering off and on; the point at which limit_req triggers should stay the same.
6. If using HTTP/2, repeat the test and verify that limit_conn does not allow more than ~50 concurrent upstream work; if it does, consider switching limit_conn to a per‑request key or lowering the connection limit.