Manual verification steps required after failed nginx master binary replacement

Direct answer

Before signaling the production master, validate the replacement binary on staging using the production configuration, alternate listen ports, and live HTTP requests. Compare build metadata between binaries. For rollback, do not trust the .oldbin PID file; verify every PID against /proc, keep the old master running until the new binary is stable, and keep a known-good copy of the previous binary outside the package-managed path.

Confirmed facts and likely explanation

Confirmed: nginx -t checks configuration syntax only. It does not verify binary ABI, dynamic-module compatibility, or shared-library symbol resolution. The USR2 upgrade signal is the standard mechanism for replacing the executable; the previous master conventionally renames its PID file to .oldbin.

Likely explanation: A package manager overwrote the binary, or the replacement was compiled with different modules or flags, causing bind or symbol failures only after the new master starts.

Staging verification before USR2

1. Copy the replacement binary to a temporary path the package manager cannot touch, for example /tmp/nginx-new/nginx.
2. Run syntax validation against the production config: /tmp/nginx-new/nginx -t -c /etc/nginx/nginx.conf. Treat a pass as necessary but not sufficient.
3. Compare nginx -V output from the current binary and the replacement. Confirm identical build options, compiler flags, and loaded dynamic modules.
4. Start the replacement on staging with alternate ports, for example 8080 and 8443, using the production configuration.
5. Send HTTP requests to those ports and inspect the nginx error log for module-load errors, unresolved symbols, or bind failures.
6. Confirm the master and worker processes remain alive after several requests.

Reliable rollback when .oldbin cannot be trusted

1. Before USR2, read the current master PID from its PID file and confirm the process is alive: ps -p "$(cat /var/run/nginx.pid)" -o pid,comm,args. Verify /proc/PID/exe points to the expected old binary.
2. Send USR2 to the current master.
3. After USR2, verify both the primary PID file and .oldbin exist, then check /proc/PID/exe for each PID. Do not signal a PID solely because it appears in .oldbin.
4. If the new master fails, stop it with TERM or QUIT. If the old master is still running, leave it active; it continues accepting connections on the production port.
5. If the old master has already exited, restart from the saved previous binary outside the package-managed path.
6. Send WINCH to the old master only after the new binary has proven stable, so old workers drain without dropping active connections.

Assumptions and uncertainty

This guidance assumes you can preserve a copy of the previous binary outside the package manager. Package-manager overwrite behavior varies by distribution. The .oldbin convention is not guaranteed across all nginx builds or custom patches. Because the supporting research is model knowledge rather than fetched source evidence, treat this as a draft for human review before applying to production.

Missing diagnostic detail

Is the previous master process still running, or do you have a saved copy of the previous binary outside the package-managed path? This determines whether rollback requires only stopping the new master or restoring and restarting the old binary.