uWSGI chain-reload limits during long-running requests

Question

uWSGI provides a chain-reload mechanism to facilitate zero-downtime updates by using the SIGHUP signal. By enabling master=true, the master process spawns new workers before terminating old ones, maintaining request continuity during code migrations.
A primary concern arises when the application handles long-running requests that exceed the configured worker timeouts. While the master waits for workers to finish current tasks before exiting, the interaction between worker-reload limits and signal-driven cycles can lead to premature termination if the state transition is not perfectly timed.
How does uWSGI manage the queue of new workers if all existing workers are busy processing long-lived requests during a reload? Is there a specific configuration to limit the maximum number of concurrent worker generations allowed during a chain-reload to prevent resource exhaustion?

Tasadduq Burney · Answer

Worker Management During Chain-Reloads
uWSGI manages the transition of workers during a chain-reload by replacing them sequentially. When a reload is triggered, the master process spawns a new worker to replace an existing one. The new worker begins accepting requests immediately, while the old worker is signaled to shut down gracefully after it completes its current task.
If all existing workers are busy with long-lived requests, the reload process does not "queue" additional workers beyond the configured processes count. Instead, the master process waits for the specific worker being replaced to finish its task before it is fully terminated. This ensures that the total number of active workers does not grow indefinitely, but it does mean that the reload cycle for a specific worker slot will stall until that worker becomes idle or is forcibly killed.

Resource Exhaustion and Generation Limits
There is no specific configuration to limit the number of "generations" of workers; however, because chain-reloading replaces workers one-by-one, the maximum memory overhead is typically limited to the cost of processes + 1. The resource risk occurs when multiple workers are stuck in a "graceful shutdown" state while their replacements are already active, effectively doubling the memory footprint for those specific slots.

Key Configuration Controls

reload-mercy: This is the critical setting for long-running requests. It defines the number of seconds the master process waits for a worker to shut down gracefully before sending a SIGKILL. If your requests take 60 seconds but reload-mercy is set to 30, the request will be terminated prematurely.
master = true: Required for the master process to manage the signal-driven reload cycle.

Verification and Safe Implementation
To ensure your reload settings accommodate your longest expected request, you can verify the behavior using a scoped test endpoint:

Create a temporary route that sleeps for a duration longer than your current reload-mercy setting.
Trigger a reload via SIGHUP.
Monitor the process list to see the overlap: ps aux | grep uwsgi.
Check the uWSGI logs for worker killed messages, which indicate that reload-mercy was exceeded.

Assumption: This behavior assumes a standard uWSGI deployment using the master process. If you are using an external process manager (like systemd) to restart the entire uWSGI stack rather than using internal chain-reloading, these internal timers will not apply.

To provide a more precise reload-mercy recommendation, could you specify the maximum expected duration of your longest request?

uWSGI chain-reload limits during long-running requests

1 answer

Worker Management During Chain-Reloads

Resource Exhaustion and Generation Limits

Key Configuration Controls

Verification and Safe Implementation

Your perspective belongs here.

0 question comments