Diagnosing uWSGI Worker Recycling and Memory Issues: A Practical Guide

Recognizable Condition

Workers restart silently, latency spikes during reload, or the upstream proxy returns intermittent 502 errors even though reload-on-rss is set. In logs you may see no error messages, yet the master process reports worker deaths or OOM kills from the kernel.

Cause/Diagnostic Table

Observed Symptom	Log Indicator	Likely Cause
Worker disappears after a fixed number of requests	worker N recycled after M requests	max-requests recycling
RSS grows but no reload log appears	No reload-on-rss message despite RSS > limit	reload-on-rss set above cgroup limit or too close to it
Requests abort with harakiri timeout	harakiri worker killed with request path	harakiri too low for actual request latency
Graceful reload takes longer than expected and workers fail to start	mercy messages followed by traceback or import errors	worker-reload-mercy shorter than app bootstrap time
Sudden OOM kill from kernel	Kernel log shows Out of memory: Kill process	reload-on-rss race: kernel OOM fires before uWSGI signal

Ordered Checks

1. Enable the stats server and inspect worker metrics.

   # Run on the host where uWSGI runs, as a user that can bind to port 1717
   stats = :1717
   memory-report = true
   

   Then, with appropriate network access (usually localhost), run:

   curl -s http://localhost:1717/workers | jq .

   Look for rising rss values, increasing requests counters, and non‑zero harakiri_count or last_spawn timestamps. Risk: Exposing the stats port to untrusted networks can leak internal metrics; bind to localhost or protect with a firewall.
2. Correlate uWSGI logs with proxy 502 responses.

   # Assuming you have journalctl access
   journalctl -u uwsgi-emperor -f | grep -E "recycled|harakiri|reload-on-rss"

   Match timestamps with your proxy’s error log (e.g., nginx error.log).
3. Verify reload-on-rss relative to the cgroup memory limit.

   # cgroup v2
   cat /sys/fs/cgroup/memory.max
   # cgroup v1
   cat /sys/fs/cgroup/memory/memory.limit_in_bytes
   

   Ensure the configured value is ≤ 80 % of the limit.
4. Confirm max-requests is non‑zero and uses jitter.

   # In your uWSGI config
   max-requests = 5000
   max-requests-delta = 1000
   

   The delta adds per‑worker randomness to avoid thundering‑herd reloads.
5. Check harakiri settings against observed latency.

   harakiri = 30
   harakiri-verbose = true
   

   Send a test request that sleeps longer than the harakiri value and verify the verbose log includes the request path and elapsed time.
6. Compare worker-reload-mercy to actual application bootstrap time.

   # Measure bootstrap (e.g., time to load imports and init)
   time uwsgi --ini yourapp.ini --single-interpreter --exit-after-init
   

   Set worker-reload-mercy to at least twice that measured time.

Fixes Tied to Findings

• If max-requests recycling is too frequent, increase the base value and add a delta to spread reloads.
• If reload-on-rss never triggers despite high RSS, lower it to 70‑80 % of the cgroup limit and enable memory-report = true for visibility.
• If harakiri fires on legitimate requests, raise the timeout to roughly 2× the observed 99th‑percentile latency and keep harakiri-verbose on to distinguish app slowness from external waits.
• If graceful reloads fail because workers die before finishing init, increase worker-reload-mercy (default 60 s in uWSGI ≥ 2.0.15) to exceed your measured bootstrap time.
• Add hard limits limit-as and limit-rss slightly below the cgroup limit to give the kernel a safety margin and prevent OOM races.

Escalation Criteria

• Worker recycling rate > 5 times per minute after applying fixes → suspect a memory leak in a C‑extension or mis‑configured limits; consider profiling with valgrind or heaptrack.
• Harikiri triggers on > 1 % of requests → enable harakiri-verbose and profile the application (e.g., cProfile, py-spy) to locate slow code paths.
• Reload‑on‑RSS fires but RSS continues to grow in newly spawned workers → likely a leak in a native module; prefer reload-on-rss over max-requests for memory‑driven recycling.
• Emperor vassal fails to spawn after three retries → check vassal‑specific config, file permissions, and ensure each vassal has its own reload-on-rss (they do not inherit the master’s limits).
• Stats server unresponsive while uWSGI appears busy → possible master block; capture a gdb backtrace of the master process (gdb -p -ex "thread apply all bt" -ex quit) and consider upgrading to a newer uWSGI version.

Example Configuration

[uwsgi]
master = true
processes = 4
socket = :3031
# Memory‑driven recycling
reload-on-rss = 250M          # adjust to ≤80% of cgroup limit
memory-report = true
# Request‑count based recycling with jitter
max-requests = 7000
max-requests-delta = 1500
# Harakiri settings
harakiri = 45
harakiri-verbose = true
# Graceful reload mercy
worker-reload-mercy = 120
# Hard limits (optional safety net)
limit-as = 260M
limit-rss = 250M
# Disable auto‑reload in production
py-auto-reload = 0
# Stats server for diagnostics
stats = :1717

Verification Steps

1. Start uWSGI with the above config (or your adapted version).
2. Confirm the version supports cgroup v2 RSS reading:

   uwsgi --version
   # Expected output: 2.0.18 or higher
   

3. Check the cgroup limit and compare:

   # v2 example
   limit=$(cat /sys/fs/cgroup/memory.max)
   echo "Limit: $limit bytes"
   reload=$(uwsgi --get-option reload-on-rss 2>/dev/null)
   echo "Configured reload-on-rss: $reload bytes"
   # Ensure reload < 0.8 * limit
   

4. Generate load (e.g., with wrk) and watch the stats endpoint:

   wrk -t4 -c200 -d2m http://localhost:3031/ &
   # In another terminal
   while true; do curl -s http://localhost:1717/workers | jq '.[] | {pid, rss, requests, harakiri_count}'; sleep 5; done
   

   Verify that rss stays below the reload-on-rss threshold and that workers are recycled according to the max-requests + delta pattern.
5. Trigger a manual graceful reload and observe mercy timing:

   kill -HUP $(pidof uwsgi)
   # Log should show "mercy" followed by worker spawn within worker-reload-mercy seconds
   

6. Validate harakiri‑verbose:

   # Send a request that sleeps 50 seconds while harakiri=45
   curl "http://localhost:3031/slow?delay=50" &
   # Check uWSGI logs for a harakiri entry with the path and elapsed time