NGINX Reverse Proxy Caching: Architecture Note on proxy_cache and cache_lock

Requirements

An API gateway sits in front of three upstream application servers. The upstream responses are expensive to generate—each request triggers a 200–500 ms database query and template render. Traffic peaks at 2,000 requests per second, with 80% hitting the same 500 URLs. The team needs to reduce upstream load by at least 70% while keeping p99 latency under 150 ms. Cache invalidation must be explicit (no TTL-only expiry), and authenticated responses must never be shared across users.

Smallest Suitable Design

Use a single proxy_cache zone backed by a shared memory index (keys_zone) and a dedicated cache partition on fast local SSD. Enable cache_lock to serialize concurrent misses for the same key. The minimal http block configuration:

http {
    proxy_cache_path /var/cache/nginx/api
        levels=1:2
        keys_zone=api_cache:100m
        max_size=50g
        inactive=24h
        use_temp_path=off;

    server {
        listen 80;
        location /api/ {
            proxy_pass http://app_upstream;
            proxy_cache api_cache;
            proxy_cache_key "$scheme$proxy_host$request_uri";
            proxy_cache_valid 200 302 10m;
            proxy_cache_valid 404 1m;
            proxy_cache_lock on;
            proxy_cache_lock_timeout 5s;
            proxy_cache_lock_age 5s;
            proxy_cache_bypass $http_authorization;
            proxy_no_cache $http_authorization;
            add_header X-Cache-Status $upstream_cache_status;
        }
    }
}

Why each directive matters:

• keys_zone=api_cache:100m — 100 MB shared memory holds ~800,000 keys (≈128 bytes/entry). Size it for your unique URL count plus 20% headroom.
• levels=1:2 — Two-level directory tree avoids hitting per-directory inode limits on ext4/xfs.
• use_temp_path=off — Writes directly to the final cache path, avoiding a cross-device rename when proxy_temp_path lives on a different filesystem.
• proxy_cache_lock on — First miss fetches from upstream; subsequent requests for the same key wait in the shared memory lock table instead of all hitting upstream.
• proxy_cache_lock_timeout 5s — After 5 seconds a waiting request gives up and goes to upstream directly. Set this to your upstream p99 + margin.
• proxy_cache_lock_age 5s — If the lock holder crashes or stalls longer than this, another request may take over the fetch.
• proxy_cache_bypass $http_authorization and proxy_no_cache $http_authorization — Any request carrying an Authorization header skips cache lookup and storage entirely, preventing credential leakage.

Trust and Data Boundaries

The cache sits in a trusted infrastructure zone: only the NGINX process and the cache manager/loader helper processes read/write /var/cache/nginx/api. Upstream applications are untrusted data producers—they must not control cache keys or TTLs via response headers unless you explicitly allow it.

Key boundary decisions:

• Cache key composition: The example uses $scheme$proxy_host$request_uri. Do not add $cookie_sessionid or $http_authorization to the key; instead use proxy_cache_bypass to skip cache for those requests entirely. Adding session identifiers to the key creates a unique entry per user, defeating the purpose of shared caching.
• Response headers: Upstream Cache-Control, Expires, and Set-Cookie are ignored unless you set proxy_cache_valid to honor them (via proxy_cache_valid any with proxy_ignore_headers). The safer default: ignore upstream caching directives and enforce TTL at the proxy layer.
• Purge/invalidation: Use proxy_cache_purge (requires ngx_cache_purge module) or rename the cache directory and reload. Do not expose purge endpoints to untrusted clients.

Operational Checks

Verify the build includes proxy module

nginx -V 2>&1 | grep -o with-http_proxy_module

If absent, rebuild or install a package with the module.

Confirm cache hit/miss behavior

# Warm the cache
curl -s -o /dev/null -w "%{http_code} %{header{X-Cache-Status}}\n" http://gateway/api/resource/123
# Repeat; second request should show HIT
curl -s -o /dev/null -w "%{http_code} %{header{X-Cache-Status}}\n" http://gateway/api/resource/123

Expected sequence: 200 MISS then 200 HIT. A request with Authorization: Bearer token should return BYPASS.

Monitor lock contention under load

# Simulate 50 concurrent requests for the same uncached URL
for i in {1..50}; do curl -s -o /dev/null http://gateway/api/resource/456 & done; wait
# Check upstream access logs: only 1–2 requests should reach upstream

Use ngxtop or awk on access logs to count upstream requests per cache key during a cache expiration event.

Cache manager/loader health

grep -E "cache (manager|loader)" /var/log/nginx/error.log | tail -20

Look for "cache manager process started" and periodic "freeing" lines. A stuck loader on startup (scanning a 50 GB cache) can delay worker readiness—consider loader_files=1000 loader_threshold=300 loader_sleep=50 in proxy_cache_path to throttle.

Failure Modes

Failure	Symptom	Mitigation
keys_zone exhaustion	New entries not cached; logs show "could not allocate node in cache keys zone"	Increase keys_zone size or reduce unique keys (normalize query strings). Monitor with nginx -T and shared memory stats.
Cache partition full	502 on cache write; "No space left on device" in error.log	Set max_size below filesystem capacity; alert at 80% usage. Use separate mount for /var/cache/nginx.
Upstream slower than cache_lock_timeout	Waiting requests time out, all go upstream, thundering herd returns	Set proxy_cache_lock_timeout ≥ upstream p99 + 2× margin. Add proxy_cache_use_stale updating to serve stale content while one request refreshes.
NGINX restart loses lock state	Brief duplicate fetches after reload/restart	Acceptable for most workloads. If unacceptable, use proxy_cache_use_stale with updating so only one worker refreshes after restart.
Authenticated response cached	User B sees User A's data	proxy_cache_bypass $http_authorization; proxy_no_cache $http_authorization; — test with two curl sessions, one with token.
Stale content served indefinitely	Upstream changed, cache never refreshes	Set inactive in proxy_cache_path and use explicit purge on deploy. Do not rely solely on TTL.

Conditions That Would Change the Design

• Multi-datacenter deployment: Add proxy_cache_key including $host or a datacenter identifier; consider a distributed cache (Redis/Memcached) via ngx_http_redis_module instead of local disk.
• Vary by Accept-Language or device type: Extend proxy_cache_key with $http_accept_language or a mapped $device_type variable. Watch keys_zone growth.
• Upstream supports conditional requests (ETag/Last-Modified): Enable proxy_cache_revalidate on; to convert expired entries to If-None-Match requests, saving bandwidth.
• Regulatory requirement for zero disk persistence: Use proxy_cache_path on a tmpfs mount; accept cache loss on reboot.
• Cache hit ratio below 40%: Re-evaluate key normalization (strip tracking params), increase inactive, or move caching to application layer where business keys are clearer.
• Upstream returns Set-Cookie on cacheable responses: Add proxy_ignore_headers Set-Cookie; only if you are certain the cookie is not session-critical, otherwise keep proxy_no_cache triggered by Set-Cookie presence.

Verification Checklist Before Production

1. Run nginx -t and confirm syntax OK.
2. Start NGINX; verify cache directory ownership (nginx:nginx or dedicated user) and permissions (700).
3. Execute the curl HIT/MISS/BYPASS test sequence above.
4. Load-test with wrk or hey at 1.5× peak; confirm upstream request count drops ≥70%.
5. Simulate cache expiry: touch -d '1 hour ago' /var/cache/nginx/api/*/*/* then load-test; verify cache_lock limits concurrent upstream fetches.
6. Enable access log $upstream_cache_status and ship to observability; alert on MISS rate spike >20% above baseline.