Nomad built-in DNS TTL and cache invalidation after allocation reschedule

DNS TTL and Configuration

Nomad's built-in DNS server uses fixed default TTL values for service and allocation records. These values are generally global and cannot be configured on a per-service basis. Because the DNS server is a lightweight utility integrated into the Nomad server, it does not support the granular record-level TTL overrides found in full-featured DNS providers like CoreDNS or BIND.

Handling Allocation Rescheduling

When an allocation is rescheduled, the Nomad server updates its internal state immediately. However, the built-in DNS server employs a passive invalidation model. It does not push updates to clients or signal the removal of records.

Clients will continue to receive the stale IP address until the TTL of the previous record expires. If the client environment employs its own caching layer (such as systemd-resolved on Linux or the JVM's internal DNS cache), the record may persist even longer than the TTL specified by the Nomad server.

Verification Steps

To determine the exact TTL currently served by your cluster and observe the propagation delay during a reschedule, use the following method:

1. Inspect the TTL: Run a query directly against a Nomad server IP to see the answer section's TTL value:

   dig @<nomad-server-ip> <service-name&>.nomad

2. Monitor Rescheduling: Trigger a reschedule (e.g., via nomad alloc stop) and repeatedly query the DNS server to measure the delta between the allocation becoming healthy and the DNS record updating.

TLS and DNS Interaction

There is no functional overlap between the TLS configuration used for the Nomad HTTP API/RPC and the built-in DNS server. DNS queries are served over standard UDP/TCP port 53 and are unencrypted. Enabling TLS for the API or configuring CA certificates for server-to-server communication does not affect DNS resolution or the validity of DNS-based service discovery.

Assumptions and Constraints

This behavior assumes a standard Nomad server deployment where the dns block is enabled. We assume the client is querying the Nomad server directly; if a recursive resolver is placed between the client and Nomad, that resolver's own caching policies will override Nomad's TTL.

Missing Diagnostic Detail: Are you using a language runtime with a known aggressive DNS cache (e.g., Java/JVM)? If so, the perceived stale record duration is likely a client-side configuration issue rather than a Nomad server TTL limitation.