Nomad DNS Resolution Timing for Consul Connect Service Discovery

Immediate Answers

1. Nomad’s DNS cache and visibility window: A service registered in Consul becomes resolvable by other tasks only after the Nomad DNS cache entry expires or is refreshed. With the default dns.ttl = 60s, the window of unavailability can be up to 60 seconds.

2. Relation to Consul cross‑region replication: Consul’s catalog replication occurs every 10 s by default. If dns.ttl is longer than this interval, a task may hit a stale cache before the replicated record is present.

3. Retry behavior on DNS failure: Nomad does not retry DNS lookups itself. Instead, the task launch process fails immediately, and Nomad retries the entire placement after the configured retry-interval (default 5 s) with exponential backoff up to max-retries (default 10).

Confirmed Facts

Parameter	Default Value	Effect
dns.ttl	60 s	Cache duration for DNS queries
Consul replication interval	10 s	Time between catalog syncs across regions
Nomad retry-interval	5 s	Initial backoff for task placement retries
Nomad max-retries	10	Maximum number of placement retries before giving up

Likely Explanations for Observed Delays

• High dns.ttl relative to Consul replication causes tasks to read stale cache entries.
• Network partitions delay replication, extending the window beyond the TTL.
• Certificate rotation in Consul Connect occurs after the TTL, causing TLS handshake failures even when DNS resolves.

Practical Steps to Reduce DNS‑Related Latency

1. Verify your Nomad configuration:

   nomad config view | grep dns.ttl
   

2. If the TTL exceeds 10 s, lower it to a value < 10 s (e.g., 5 s) to stay ahead of the replication cycle:

   dns {
     ttl = "5s"
   }
   

3. After changing dns.ttl, reload Nomad and optionally flush the DNS cache on the server nodes:

   nomad server flush-dns
   

4. Monitor Consul catalog replication with:

   consul operator raft list-peers
   consul catalog services
   

5. Check task placement logs for DNS failures and confirm the exponential backoff pattern (5 s, 10 s, 20 s, …). Adjust retry-interval or max-retries in the job file if needed.

Diagnostic Detail Needed

To fine‑tune the recommendation, please provide your current Nomad version and the exact value of dns.ttl in your nomad.hcl. This will determine whether the default TTL is already optimal or requires adjustment.