Does nginx support atomic rollback of upstream configuration after reload
0 reputation · 17 Dec 2024, 16:01 UTC
0 reputation · 17 Dec 2024, 16:01 UTC
The goal is to change upstream definitions in nginx and be able to revert safely if the new configuration is not viable, without leaving stale backend references or interrupting in-flight requests.
Reload via SIGHUP is documented to replace worker processes while preserving existing connections and applying new configuration. A syntax error in the new configuration can cause workers to exit, requiring manual restoration of the previous configuration. The include directive enables modular configuration files, but inter-dependencies between included blocks can produce unexpected behavior when a new configuration references directives that are removed on rollback. Zero-downtime changes are discussed for log reopening, while core configuration changes are treated differently.
An unresolved design question remains around atomic configuration swaps without service interruption, a capability that has been requested in community discussions but is not present as a stable documented feature. Does nginx provide a built-in mechanism to revert to the previous configuration automatically if a reload fails? Which guarantees exist for existing connections when upstream groups are renamed or removed? When is a full restart required versus reload for changes to upstream definitions?
31275 reputation · 17 Dec 2024, 22:53 UTC
nginx does not provide a built-in automatic rollback mechanism for upstream configuration changes after a reload. When you send a reload signal (SIGHUP), the master process validates the new configuration. If validation fails, the new worker processes are never started and the existing workers continue serving with the previous configuration — this is a passive safety net, not an active rollback. If validation succeeds but the new upstream definitions cause runtime problems (unreachable backends, routing errors), nginx will not revert automatically.
On successful reload, nginx starts new worker processes with the updated configuration while old workers continue processing in-flight requests. Old workers exit only after their connections complete or the worker_shutdown_timeout is reached. New connections immediately use the new configuration.
Adding, removing, or modifying servers within an upstream block, changing parameters (weights, max_fails, etc.), or renaming upstream groups are all handled via reload. If an upstream group referenced by active connections is removed or renamed, those connections continue on the old workers until they naturally terminate. The old group name remains in memory for the lifetime of those workers.
A full stop/start cycle is only necessary for changes affecting the master process itself: user/group directives, master-level settings (pid file, worker_processes count), or certain SSL/TLS parameters requiring master reinitialization. Upstream definition changes never require a full restart.
Since nginx lacks transactional configuration swaps, implement rollback at the deployment layer:
nginx -t# Test configuration syntax
nginx -t
# Trigger reload
nginx -s reload
# Observe process transition
ps aux | grep nginx
# Check error log for validation failures
tail -f /var/log/nginx/error.logEstablish a long-lived connection (curl --no-buffer -N), trigger reload, and verify the connection completes on the old worker.
What nginx version are you running? worker_shutdown_timeout behavior and reload semantics have evolved — run nginx -V to confirm your version's capabilities.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.