Question
NATS DNS SRV resolution failure: client behavior when SRV query yields no records
Nia Echo
0 reputation · 08 Sept 2021, 08:10 UTC
34.2K views1
Loading…
0 reputation · 08 Sept 2021, 08:10 UTC
When a NATS client is configured to discover servers via DNS SRV records, the documentation does not state what occurs if the SRV query returns no results or encounters an error. In practice, logs rarely contain explicit DNS error messages, making it hard to determine whether a failure is due to network issues or a mis‑configured SRV record. Additionally, the client’s TLS verification mode, controlled by the tlsverify flag, can influence whether a connection attempt is retried or aborted when a server certificate cannot be validated against the system’s trusted CA store. The lack of a documented retry policy for transient DNS failures creates uncertainty about the client’s resilience and recovery behavior.
Unresolved decisions include whether the client should automatically retry DNS SRV resolution after a transient failure, and if so, what backoff strategy should be applied. The interaction between tlsverify and DNS resolution failures also remains unclear.
1. Does the NATS client automatically retry DNS SRV resolution after a transient failure, and what backoff strategy does it use, if any? 2. How does enabling tlsverify affect the client’s retry logic when DNS SRV resolution fails or returns no records?
35275 reputation · 08 Sept 2021, 19:35 UTC
1. Automatic DNS SRV retry – The official NATS Go client (v2.x) does not treat a missing SRV response as fatal. It keeps the last‑known server list (or the static list supplied in Options.Servers) and starts a periodic SRV re‑lookup. The retry back‑off begins at 1 s, doubles each attempt, and caps at 30 s. After reaching the cap, the client continues to query every 30 s until a record is returned.
2. tlsverify interaction – The tlsverify flag only influences the TLS handshake after an address has been resolved. It does not alter the SRV retry schedule. If a server certificate cannot be validated, the connection attempt is aborted, but the DNS retry loop continues unchanged. When tlsverify is false, the client will still try to connect to the resolved address even if the certificate is invalid, and any TLS failure is handled by the normal reconnect logic, independent of the DNS loop.
Because the client treats SRV failures as transient, a single empty response or temporary DNS outage does not stop the client from attempting to connect. The back‑off mechanism prevents DNS storms while still giving the client a chance to recover quickly once the DNS service is healthy again. The separation of DNS resolution and TLS validation keeps the retry logic simple and predictable across different TLS configurations.
Run the client with debug logging enabled (e.g., nats -DV or set nats.Options.Verbose = true in Go). Look for log lines that start with Resolving SRV or SRV lookup failed.
Point the SRV target to a domain that returns no records (or temporarily block DNS responses using iptables or a local DNS server). Confirm that the client logs a retry after 1 s, then 2 s, 4 s, etc., up to 30 s.
Restore DNS resolution and verify that the client logs another SRV lookup and immediately connects once a record appears.
Toggle tlsverify (set to true and false) while using a self‑signed server. Observe that TLS failures are reported separately and do not affect the SRV retry cadence.
Do you know which language client and version you are using? The behavior described above applies to the Go client v2.x; other language bindings (Java, JavaScript, Python, etc.) may implement different retry logic.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.
nats