Consul DNS for Service Discovery: Setup, Verification and Rollback

If your applications hardcode backend hostnames, adding or replacing an instance means editing config and restarting. Consul's DNS interface removes that step: a client resolves web.service.consul and receives the addresses of instances that are currently passing their health checks. The catch is that this only works when two things are true — the service has a health check, and the host resolver actually sends .consul queries to the agent. This guide covers both, plus how to prove the behavior and how to undo it.

Desired outcome

After working through this, you should be able to:

• Register a service with a health check.
• Resolve it directly against the agent's DNS listener on port 8600.
• Resolve it from a normal host lookup without specifying the port.
• Confirm that a failing instance stops appearing in answers.

Prerequisites and version assumptions

• A Consul agent reachable from the machine doing the lookup. The behavior described here matches the Consul 1.x DNS interface; verify the specifics against the documentation for your exact version before rolling out (flagged for review).
• The agent's DNS listener on its default port, 8600. If ports.dns was changed in the agent config, adjust the queries below.
• dig or nslookup installed for testing.
• Root or sudo to edit the host resolver configuration, if you want plain hostname lookups.
• If ACLs are enabled, a token with permission to read the service. How ACLs apply to the DNS interface is agent-side and version-dependent — confirm this for your release rather than assuming.

Step 1: register a service that can actually fail

The DNS interface filters on health. A service registered with no check has nothing to fail, so Consul treats it as healthy indefinitely — this is the most common reason a dead instance keeps resolving. Define a check explicitly.

{
  "service": {
    "name": "web",
    "port": 8080,
    "tags": ["v1"],
    "check": {
      "id": "web-http",
      "name": "web HTTP check",
      "http": "http://127.0.0.1:8080/health",
      "interval": "10s",
      "timeout": "2s"
    }
  }
}

Run this on the node hosting the service, where the local agent listens on the default HTTP port:

consul services register web.json

This writes to the local agent's catalog, so it needs write permission for that service (with ACLs enabled, a token carrying service write). The interval value controls how quickly a failure is noticed; the DNS answer can be stale for up to roughly that long.

Step 2: query the agent directly before touching the resolver

Always test against port 8600 first. It isolates Consul problems from resolver problems.

dig @127.0.0.1 -p 8600 web.service.consul A
dig @127.0.0.1 -p 8600 web.service.consul SRV
dig @127.0.0.1 -p 8600 v1.web.service.consul A

What to look at in the response:

• Answer section — expect one record per passing instance. If nothing comes back, either no instance is passing or the name is wrong.
• SRV records — the target and port fields. This is the record type to use when instances get dynamic ports, because an A record carries only an address.
• TTL field — Consul's DNS responses are designed to be re-queried frequently. Whether the TTL is zero or a configured value depends on dns_config in your agent config; check the returned TTL rather than assuming it.

Tagged lookups use the form <tag>.<service>.service.consul, which is useful for canary routing without changing the client.

Step 3: make plain host lookups work

Two patterns are common. Pick one; mixing them usually causes confusing resolution order.

Option A: point the host at Consul and add a search domain

search consul
nameserver 127.0.0.1

With this, web.service resolves without the suffix. The important consequence: all lookups now go to Consul. Consul forwards names it does not own to the upstream servers listed in its recursors setting. If recursors is not configured, non-Consul lookups will fail. Confirm that setting before switching a host over.

Option B: forward only the .consul domain

Keep your existing resolver and delegate one domain. With dnsmasq, a line like this forwards .consul to the agent's non-standard port:

server=/consul/127.0.0.1#8600

This is the safer option on shared hosts. Note that systemd-resolved does not support forwarding to a non-standard port, so on distributions using it you would need a forwarder such as dnsmasq or unbound, or Consul listening on port 53. Verify the resolver behavior for your distribution and version.

Also be aware that /etc/resolv.conf is frequently rewritten by NetworkManager or systemd-resolved. Edit it through the distribution's mechanism, or your change will disappear at the next network event.

Expected checks

1. Count matches. The number of addresses returned should equal the number of instances passing their checks. Compare against the agent's own view before trusting DNS.
2. Ports are correct. For dynamic-port services, confirm the SRV response carries the real listening port, not a default.
3. Failure removes the instance. Put the service into maintenance mode on one node, which marks its checks critical, then re-query:

consul maint enable -service web
# re-run the dig commands from Step 2
consul maint disable -service web

Older releases use consul maint -enable -service web with the flag form; check consul maint -h on your version. The failing instance's address should be absent from the answer while maintenance is on, and should return after it is disabled.

1. Caching is not hiding the change. If the instance still appears after maintenance is enabled, the answer is being cached somewhere — the OS resolver, a local caching daemon, or the application's own DNS cache. Test with dig directly against port 8600 to bypass the host resolver and narrow it down.

Recovery and rollback

This procedure changes state in three places, so each has its own undo:

• Resolver config: restore the previous resolv.conf (or forwarder stanza) from a backup and restart the resolver. Keep a copy before editing, since a broken resolver affects package installs and other tooling.
• Maintenance mode: consul maint disable -service web. Leaving a service in maintenance mode silently removes it from DNS answers.
• Service registration: consul services deregister -id web removes the definition from the local agent. Deregistering removes it from DNS answers immediately, so confirm nothing still depends on the name.

Limitations worth knowing before you commit

Record type	Returns	Use when
A	Instance addresses only	Port is fixed and known to the client
SRV	Target host and port	Ports are dynamic, or you need per-instance ports

• DNS returns a set, not a load-balancing policy. Which address a client tries first is up to the client and resolver.
• TTL and client-side caching determine how long a failed instance can still be tried. Short intervals plus aggressive caching is a bad combination.
• ACL handling for the DNS interface is configured on the agent, not per query. If lookups return nothing after enabling ACLs, that is the first thing to check — and to confirm against your version's documentation.
• Consul's DNS interface is not a general-purpose recursive resolver. Treat upstream forwarding as a separate concern.

What to re-verify on your own deployment

Three items in this guide are version- or environment-dependent and should be confirmed rather than assumed: the default DNS response TTL and how dns_config changes it, whether your host resolver can forward to port 8600, and how ACLs gate DNS answers in your Consul release. Each is checkable in a few minutes with the dig commands above and the agent's own service listing.