Using NGINX limit_req_module for Per‑IP Rate Limiting: An Architecture Note

Problem Statement

\n

When an upstream API or web application is exposed to the Internet, it is common for attackers to issue high‑volume requests that exhaust resources, trigger credential‑stuffing, or scrape content. A simple and fast first line of defense is per‑IP request‑rate limiting. NGINX’s limit_req_module provides this capability without adding external state or latency, but its effectiveness depends on careful design and operational oversight.

\n\n

Requirements

\n

\n
• Defend downstream services from abusive traffic while keeping latency low for legitimate users.
\n
• Enforce a request‑per‑second (RPS) Service Level Agreement (SLA) per client IP.
\n
• Allow short bursts that accommodate normal user spikes but reject sustained over‑load.
\n
• Provide observable metrics (e.g., 503 responses, current request rate) for monitoring and alerting.
\n
• Support dynamic adjustment of limits without downtime.
\n

\n\n

Smallest Suitable Design

\n

The minimal configuration uses a shared memory zone to count requests per IP and applies the limiter in the server or location block that proxies to the upstream. The zone is defined once, then referenced wherever needed.

\n

# Define a shared memory zone named 'one' with 10 MB capacity.\nlimit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;\n\n# Apply the limiter to all requests reaching the API.\nserver {\n    listen 80;\n    server_name api.example.com;\n\n    location / {\n        # Allow a burst of 20 requests and reject over the limit immediately.\n        limit_req zone=one burst=20 nodelay;\n        proxy_pass http://upstream_api;\n    }\n}\n

\n

Explanation of key directives:

\n

\n
• $binary_remote_addr stores the client IP in binary form, reducing memory usage.
\n
• rate=10r/s sets the baseline RPS per IP.
\n
• burst=20 allows up to 20 queued requests before the limit is enforced.
\n
• nodelay denies queued requests when the burst is exceeded, returning HTTP 503 immediately.
\n

\n\n

Trust & Data Boundaries

\n

The module uses $remote_addr (or $binary_remote_addr) after any real_ip processing. Therefore:

\n

\n
• If NGINX sits behind a trusted reverse proxy that sets X-Forwarded-For or True-Client-IP, configure set_real_ip_from and real_ip_header before the limiter.
\n
• When no trusted proxy is present, the client’s source IP is taken directly from the TCP connection. Spoofed IPs cannot bypass the limiter under normal TCP/IP rules.
\n
• In NAT or shared‑IP environments, many users share a single public IP, which may cause legitimate traffic to hit the limit. Combine with authentication or API keys for finer granularity.
\n

\n\n

Operational Checks

\n

To ensure the limiter behaves as intended, monitor the following:

\n

\n
1. HTTP 503 Count: 503 responses are a direct indicator of limit enforcement. Use $limit_req_status in the access log:\n

   log_format main '$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $limit_req_status';\naccess_log /var/log/nginx/access.log main;\n

\n
2. Stub Status: Enable stub_status to view Reqs/sec and Active counters. Example:\n

   location /nginx_status {\n    stub_status;\n    access_log off;\n    allow 127.0.0.1;\n    deny all;\n}\n

\n
3. Prometheus Exporter: The nginx-prometheus-exporter exposes metrics like nginx_requests_total and nginx_503_total for alerting.
\n
4. Periodic Review: Export the zone’s internal state (via limit_req_zone_dump if compiled with the module) or parse logs to compute average RPS per IP. Adjust rate and burst accordingly.
\n

\n\n

Failure Modes & Design Change Conditions

\n

\n
• IP Spoofing: Attackers can circumvent per‑IP limits by sending packets from spoofed addresses if NGINX is exposed directly to the Internet without a firewall. Mitigation: place NGINX behind a hardened firewall or use a cloud load balancer that validates source IPs.
\n
• Shared IPs: In corporate networks or mobile carriers, many users share a single public IP. Legitimate traffic may be throttled. Mitigation: combine rate limiting with user‑level controls (API keys, OAuth tokens) and set a higher burst or lower rate for such environments.
\n
• Burst Misconfiguration: Setting burst too low can cause legitimate bursts to be rejected, while a high burst may allow a queue that increases latency. Monitor 503 rates and adjust gradually.
\n
• State Loss on Reload: Reloading NGINX (nginx -s reload) preserves the zone state, but a full restart clears it, causing all clients to start unthrottled. Use reloads for configuration changes, not restarts.
\n
• Memory Exhaustion: The zone size (10m in the example) limits the number of tracked IPs. If the traffic has millions of unique IPs, the zone will grow until it hits the memory cap, potentially causing NGINX to drop new entries. Increase the zone size or rotate the limiter per subnet.
\n
• High Latency Burst: With nodelay disabled, queued requests wait in the kernel until the rate is available, which can increase latency. If the application is latency‑sensitive, keep nodelay enabled and monitor latency spikes.
\n

\n\n

When to Re‑Design

\n

Consider a more sophisticated limiter if:

\n

\n
• Attack traffic originates from a small set of IPs that can be blocked via firewall rules.
\n
• You need per‑user or per‑API‑key limits instead of per‑IP.
\n
• You require dynamic, per‑environment limits (e.g., higher limits for internal traffic).
\n
• You want to integrate with an external rate‑limit service (e.g., Envoy, Cloudflare).
\n

\n

In such cases, replace or supplement limit_req_module with a distributed rate‑limiting system or use NGINX Plus’s limit_req_zone backed by Redis.

\n\n

Practical Verification Checklist

\n

\n
• Deploy the configuration in a staging environment.
\n
• Run a controlled burst test (e.g., wrk -t2 -c100 -d30s http://staging.example.com/) and verify that requests beyond the configured rate + burst receive HTTP 503.
\n
• Check the access log for $limit_req_status=503 entries.
\n
• Review the error log for messages like limiting requests, excess: ... by zone.
\n
• Confirm that a nginx -s reload immediately applies new limits without dropping existing connections.
\n
• Set up Prometheus alerts for nginx_503_total exceeding a threshold.
\n

\n\n

Conclusion

\n

NGINX’s limit_req_module is a lightweight, stateful mechanism for per‑IP rate limiting that can be deployed quickly and tuned over time. By understanding the trust boundaries, monitoring key metrics, and anticipating failure modes, you can protect upstream services with minimal overhead while keeping legitimate traffic responsive.