TTL Check Heartbeat Frequency Limits in Consul
0 reputation · 02 Apr 2024, 11:50 UTC
0 reputation · 02 Apr 2024, 11:50 UTC
In Consul, TTL checks require the service to send periodic heartbeats to keep the check in a passing state. If the heartbeat is not received within the TTL window, the check automatically transitions to critical. This mechanism shifts the monitoring responsibility from the Consul agent to the service itself.
When many services register TTL checks, the aggregate heartbeat traffic can increase load on the local agent’s RPC interface. Network partitions between a service and its agent can also cause false positives, as the agent may not receive heartbeats even though the service is healthy.
Additionally, the deregister_critical_service_after setting can automatically remove services that remain in a critical state for a defined duration, but its interaction with the TTL window is not fully documented.
deregister_critical_service_after setting affect the transition from passing to critical when heartbeats are intermittently missed?32275 reputation · 02 Apr 2024, 15:42 UTC
There is no documented per-service maximum heartbeat frequency in Consul. For a TTL check, the TTL value is a deadline, not a heartbeat schedule. One heartbeat per TTL window is sufficient; send it early enough to tolerate one missed interval. There is no supported bulk or batched TTL heartbeat API—each check must be updated individually by its check ID. The deregister_critical_service_after setting does not affect the transition from passing to critical; that transition is controlled solely by the TTL. The deregistration timer starts only after the check is already critical.
The likely explanation for agent load is that every heartbeat is a local check update, and agent resource use scales with the number of services multiplied by the update rate. However, Consul documentation does not publish a numeric cap per service. Unchanged passing heartbeats should be cheaper than status or output changes, but that distinction is version- and implementation-sensitive.
Confirmed behavior: the TTL timer is reset by each update for that specific check ID. A delayed batch does not share a deadline because each update resets only its own check's TTL. Looping over many checks still produces N individual requests.
Interval on a TTL check; the TTL itself defines the timeout window.TTL/3 so two missed beats still leave margin before the deadline.deregister_critical_service_after longer than your worst-case transient outage to avoid destructive removal during flaps.If a heartbeat arrives before the TTL expires, the check stays or returns to passing and deregistration is not armed. If the check is already critical, a passing heartbeat before deregister_critical_service_after elapses should clear the critical state and cancel deregistration. Confirm this cancellation behavior for your exact Consul version.
Use the agent HTTP API to inspect and update a single TTL check. Replace 8500 with your agent HTTP port and service:redis1 with your check ID.
# Update a TTL check (heartbeat)
curl -X PUT http://127.0.0.1:8500/v1/agent/check/update/service:redis1 -d 'Status=passing'
# Inspect the check state
curl http://127.0.0.1:8500/v1/agent/checks | jq '.["service:redis1"]'
What is your exact Consul version? Endpoint names and whether DeregisterCriticalServiceAfter is expressed on the check or the service registration vary across versions, and that changes the automation you should write.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.