NGINX Stream Module ↔︎ TCP Backend Servers – Health‑Check Interoperability

Answer

In the NGINX Open Source Stream module you have two practical ways to achieve automatic upstream failover for TCP backends:

1. Passive failure detection using the built‑in max_fails and fail_timeout parameters on each server line.
2. Active health‑check via an external script that periodically tests the TCP port and updates the upstream configuration (e.g., toggles the down flag) before reloading NGINX.

Confirmed facts

• The Stream module in the open‑source edition does not provide an active health_check directive; that feature exists only in NGINX Plus.
• When a client connection fails, NGINX marks the server as unavailable after max_fails consecutive failures and keeps it in that state for fail_timeout seconds.
• An external script can change a server’s status by editing the nginx.conf (or an included file) to add or remove the down flag on the relevant server line and then issuing nginx -s reload.
• Reloading NGINX with -s reload is zero‑downtime for existing connections; worker processes finish serving current requests before shutting down.

Likely explanation & trade‑offs

Passive detection (max_fails/fail_timeout)

• Reliability: Detects problems only after a client experiences a failure. If your traffic is low or bursty, a backend could be down for up to fail_timeout seconds before NGINX stops sending new connections.
• Maintenance: Zero extra components; just tune two numbers per upstream. No risk of script‑induced misconfiguration.
• Best for: Environments where brief periods of failed connections are acceptable and you prefer a pure‑configuration solution.

Active check via external script

• Reliability: Can detect a dead backend immediately after the script’s probe interval (e.g., every 5 s) and remove it before any client sees an error.
• Maintenance: Requires a reliable monitoring script, proper permissions to edit NGINX config, and a safe reload mechanism. Errors in the script or reload can cause service disruption.
• Integration: Works with the Stream module because the down flag is honored by the same load‑balancing logic that uses max_fails. No need for Plus‑only directives.
• Best for: Setups where you need fast failover and can tolerate the operational overhead of a health‑checker.

Steps to implement the external‑script approach

1. Create a script (bash, Python, etc.) that:
   • Attempts a TCP connect to each backend (nc -z host port or equivalent).
   • If the connect fails, ensures the corresponding server line in the upstream block contains down; if it succeeds, ensures down is removed.
   • Writes the updated configuration to a temporary file, checks it with nginx -t, and if valid, runs nginx -s reload.
2. Run the script via a process supervisor (systemd, supervisord, or cron) at your desired interval (e.g., every 5 seconds).
3. Make sure the script runs as a user that can read/write the NGINX config directory and send signals to the master process.
4. Monitor NGINX error logs for reload messages and for any script‑generated errors.

When to ask for a missing diagnostic detail

The recommendation between passive and active methods hinges on whether you can tolerate the brief pause that occurs during an nginx -s reload. If your environment cannot accept even a momentary reload (e.g., strict zero‑downtime requirements with long‑lived TCP sessions), the passive method may be the only safe option unless you adopt a more complex solution like binary upgrade or a dynamic‑upstream module.

Missing detail to clarify: Do you have a maintenance window or can you accept the short reload pause caused by nginx -s reload? If the answer is “no”, the passive max_fails/fail_timeout approach is preferable; otherwise, the external script method can be used for faster failure detection.