Dynamic Endpoints Without an SDK: Consul Service Discovery and Health Checking in Practice

The problem with static endpoint lists

In any environment where instances scale horizontally—containers, VM autoscaling groups, blue-green deployments—a configuration file full of IP addresses is a liability. The moment a node is replaced, clients that cached the old address start failing. You need a mechanism that answers two questions in real time: “Which instances exist for this service?” and “Which of them can actually serve traffic right now?”

Consul’s answer: a catalog plus DNS

Consul approaches this with a service catalog maintained by agents. When a service starts, it registers itself with the local Consul agent, usually through the HTTP API or a configuration file. The agent forwards that registration to the Consul servers, which store it in the catalog. Anti-entropy mechanisms then continuously reconcile the agent’s local view with the server-side catalog, so the actual state eventually matches the desired state.

Clients do not need a Consul-specific SDK. They can use ordinary DNS. A service named api is reachable at api.service.consul. The Consul agent listens on port 8600 and answers these queries using only the instances whose health checks are currently passing.

Health checks decide what “healthy” means

Registration alone is not enough. A process can be running but unable to serve requests—deadlocked, waiting on a broken dependency, or still warming up. Consul supports several check types: HTTP, TCP, gRPC, script-based, and TTL checks. Each check is evaluated by the agent, and only services whose checks are passing appear in DNS responses.

The distinction between liveness and readiness matters. A liveness check asks, “Is the process alive?” A readiness check asks, “Can this instance accept traffic now?” If you use the same aggressive check for both, you may remove an instance that is alive but temporarily busy, causing unnecessary failover. Conversely, a lax check leaves broken endpoints in rotation.

Worked example: register, resolve, fail

The following example runs on a node with a development-mode Consul agent. Development mode is fine for local verification but is not suitable for production clusters.

Register a service named api on port 8080 with a TTL check. A TTL check is healthy as long as the service (or a sidecar) periodically calls the pass endpoint; if the heartbeat stops, the check transitions to critical after the TTL expires.

curl --request PUT \
  --data '{"Name":"api","Port":8080,"Check":{"CheckID":"api-ttl","TTL":"30s"}}' \
  http://127.0.0.1:8500/v1/agent/service/register

Required permissions: the command runs against the local agent HTTP API on 127.0.0.1:8500. If ACLs are enabled, the request needs a token with service:write permission for the api service.

Send an initial heartbeat so the check starts in a passing state:

curl --request PUT http://127.0.0.1:8500/v1/agent/check/pass/api-ttl

Query the DNS interface from the same host:

dig @127.0.0.1 -p 8600 api.service.consul +short

Expected check: the answer section should contain the registered address and port information. If the service is not yet synced, retry after a few seconds.

Now simulate failure by simply not sending further heartbeats. After the 30-second TTL elapses, the agent marks the check critical and removes the instance from DNS. Run the dig command again.

Expected check: the response should no longer include the api instance. Depending on whether other healthy instances exist, you may receive an empty answer or an NXDOMAIN response.

Risks: development mode stores data in memory and disables many safety features. In production, ensure the agent HTTP interface is bound to a trusted network and protected by ACLs.

To revert the change and remove the registration entirely, call the deregister endpoint:

curl --request PUT http://127.0.0.1:8500/v1/agent/service/deregister/api

The trade-off: freshness versus overhead

The hardest part of Consul service discovery is not the API; it is tuning. Two forces pull against each other:

• DNS TTL and client caching. Consul can return a low TTL with DNS answers, but if the operating system or application resolver caches the result longer, clients will keep using stale endpoints. You must verify that your client stack honors the TTL.
• Health-check frequency. Checking every second catches failures quickly but multiplies network and CPU load across a large cluster. Checking every 30 seconds reduces overhead but widens the window during which traffic can reach a failed node.

Read consistency adds another dimension. Consul’s default reads favor lower latency and may return slightly stale data. Consistent reads guarantee the most recent state but require a round-trip to a quorum of servers, increasing latency. For endpoint discovery, most teams accept the default and rely on health checks to filter out bad instances.

What to do next

If you are evaluating Consul, do not start with a multi-datacenter deployment. Run a single development agent, register a dummy service with a TTL check, and use dig to watch it appear and disappear. Then measure two things in your own environment: how long your clients actually cache DNS, and how much overhead your check intervals generate at your target cluster size. Those two numbers will tell you whether Consul’s DNS-based discovery fits your latency budget.

Finally, confirm the exact flags, default TTLs, and API paths against the Consul version you plan to deploy; the behavior described here reflects the general 1.x agent model, but defaults have evolved across releases.