Choosing Consul Health Checks for Ephemeral Services: A Decision Guide

Ephemeral service instances that appear behind NAT or inside short-lived containers create a discovery problem: callers need to fail fast when an instance is unhealthy, but a health check that is too aggressive will flap during startup, and a check that is too slow will keep bad instances in rotation. The useful takeaway is that Consul does not guess health for you; you must decide which component owns the health signal and how quickly an unhealthy instance must disappear from DNS and HTTP responses.

Decision: register locally, then choose the check owner

Register every instance with its local Consul client agent, not a remote server. The agent runs on the same host or pod, so it can probe loopback interfaces without crossing NAT. Gate discovery on health checks so that DNS and the HTTP API return only instances in the passing state. The remaining decision is the check type, which depends on two constraints: whether the application or the agent is better positioned to know health, and the maximum time you can tolerate before a bad instance is removed.

Comparing supported check types

Consul supports several check definitions. The table below summarizes the practical differences for ephemeral workloads.

Check type	Who owns the signal	Best for	Main risk
HTTP	Agent pulls from the app	Readiness and liveness when the app can expose a shallow endpoint	False negatives during deploys if the endpoint checks deep dependencies
TCP	Agent pulls from the OS	Cheapest port reachability probe	Misses application logic errors; the port can accept connections while the process is stuck
Script	Agent runs a local command	Maximum signal when no HTTP endpoint exists	Requires plugin availability and ACL review on every client; increases packaging and security burden
TTL	Application pushes heartbeats	Cases where only the app knows its own health	A crashed process stops heartbeating and must still be reaped; not safe for processes that legitimately pause
Alias	Mirrors another check	Sidecars that should share a primary service's health	Couples lifetimes and hides independent failure modes

Trade-offs: probe noise versus heartbeat ownership

Pull-based checks (HTTP, TCP, script) are simple because the agent owns the schedule. They add a small amount of load and can produce false negatives when an application is starting up or garbage collecting. For ephemeral instances, that flapping is costly: clients see the service appear and disappear, and connection pools churn.

Push-based TTL checks invert ownership. The application must heartbeat through the local agent before the TTL expires. This avoids probe noise, but it shifts responsibility: if the process crashes, the check goes critical and the instance is eventually removed. Choose a TTL roughly three times the heartbeat interval, and alert on the warning state before the check turns critical.

Script checks maximize signal but execute on every client, so they need plugin availability and an ACL and security review. Prefer HTTP or TCP when the same signal is available over the network. Alias checks reduce duplicate probes but couple the sidecar's lifetime to the primary service, which is rarely correct for independent failure modes.

Implementation: a shallow TCP check and a TTL check

The following pull-based example registers a service named web on port 8080 with a TCP check against loopback. The interval and timeout fields control probe cadence. deregister_critical_service_after bounds the time a long-critical instance remains registered before the agent removes it.

service {
  name = "web"
  port = 8080

  check {
    name = "tcp health"
    tcp = "127.0.0.1:8080"
    interval = "10s"
    timeout = "2s"
    deregister_critical_service_after = "30m"
  }
}

For application-level signal, replace the tcp field with an HTTP check against a local /health endpoint that reports only the process's own readiness, not the status of every downstream dependency. Deep dependency checks make discovery flap during partial outages.

The push-based example below declares a TTL check stanza. In a full service definition this block sits alongside the service metadata.

check {
  id   = "web-ttl"
  name = "app liveness"
  ttl  = "15s"
}

Run these definitions on the instance itself, either through the agent configuration directory or the service registration HTTP API. The agent must be able to bind to the loopback address shown in the tcp field.

Validating that only healthy instances answer

In a sandbox, start a development agent on the instance host. No special permissions are required for -dev mode, but production agents need ACLs that permit service registration.

# Run on the instance host or sandbox VM
consul agent -dev

Register one passing service and one failing service, then query Consul DNS on the default port 8600.

# Run on the same host as the agent
dig @127.0.0.1 -p 8600 web.service.consul

Expected behavior: the failing instance is absent from the answer, and the passing instance appears. The HTTP health API should mark the failing instance critical.

# Run on the same host as the agent
curl http://127.0.0.1:8500/v1/health/service/web?passing=true

Flip the failing endpoint to healthy and re-query. The instance should reappear within the check interval plus a short agent sync delay. To test TTL behavior, stop the heartbeat and observe the state move to critical, then confirm that deregister_critical_service_after eventually removes the registration entirely.

Limitations and version-sensitive behavior

These examples assume the default agent DNS port 8600 and long-standing HCL check fields. Consul defaults, deprecations, and the Connect/Dataplane sidecar model vary by version; verify the exact deployed build before codifying configuration.

Stub resolvers and Consul DNS caching can serve stale answers. Keep Consul DNS TTLs low and avoid aggressive client-side caching during failover tests. A TTL check is not safe for processes that may legitimately pause longer than the TTL. Finally, deregister_critical_service_after permanently removes the service registration; during network partitions or long garbage-collection pauses this can cause churn and re-registration, so set the bound generously enough to avoid removing instances that are merely slow.