Diagnosing Stale Consul Service Registrations: Why Dead Instances Stay in the Catalog

Recognizable Condition

Consul's catalog or DNS interface continues to return IP addresses for service instances that have already terminated. Consumers route traffic to pods or VMs that no longer exist, causing timeouts or errors. In the UI, the service may show as "critical" for an extended period, or a node that was shut down remains listed in consul members with a failed status.

Cause and Diagnostic Table

Observed Symptom	Likely Root Cause	Quick Verification
Service stays "critical" indefinitely after process death	Registration missing DeregisterCriticalServiceAfter (or value too large)	Check the service definition: consul catalog service -service= -format=json and look for the field.
Health check never flips to critical	Check definition mismatch: wrong HTTP path, port, or TTL check not updated by the application	Run consul agent checks on the node; examine Output and Status fields.
Check flaps between passing and critical	Agent cannot reach the service endpoint (network policy, firewall, wrong bind address)	From the agent host, curl -v http://:/health using the same parameters as the check.
Dead node persists in consul members and catalog	Node left the Serf cluster ungracefully (SIGKILL, power loss) so the agent never ran deregistration	consul members -wan shows the node with status "failed" for longer than the Serf suspicion timeout.
Stale entries reappear after manual deregistration	Anti‑entropy blocked because the agent lost contact with servers (network partition, ACL token expiry)	Check agent logs for "anti‑entropy" sync errors; verify consul operator raft list-peers shows healthy quorum.

Ordered Diagnostic Checks

1. List failing service instances — Run on any machine with HTTP access to the Consul API (token with service:read):

   curl -H "X-Consul-Token: $TOKEN" \
     "https://consul.example.com/v1/health/service/my-service?passing=false"
   

   Look for entries with Status: critical and note their Node and ServiceID.
2. Inspect local agent state — On the node that hosts the stale instance (requires local shell or SSH):

   consul agent checks          # shows all checks and their current status
   consul agent services       # shows registered services with their definitions
   

   Verify whether the check is actually executing (timestamp recent) and whether the service definition includes DeregisterCriticalServiceAfter.
3. Check cluster membership — From a server or any agent with node:read:

   consul members -detailed
   consul catalog nodes
   

   Nodes stuck in "failed" state for more than a few minutes indicate ungraceful departure.
4. Review agent logs — On the problematic node (usually /var/log/consul/ or journalctl):

   journalctl -u consul -f | grep -E "check|sync|anti-entropy|deregister"
   

   Look for check execution errors, failed anti‑entropy syncs, or missing deregistration calls.

Fixes Mapped to Findings

1. Add or Adjust DeregisterCriticalServiceAfter

When registering a service via configuration file or API, include a reasonable timeout (e.g., 30s–5m). Example service definition (JSON):

{
  "service": {
    "name": "web",
    "port": 8080,
    "check": {
      "http": "http://localhost:8080/health",
      "interval": "10s",
      "timeout": "2s"
    },
    "deregister_critical_service_after": "60s"
  }
}

Apply with consul services register web.json (requires service:write). The agent will automatically deregister the service 60 seconds after its check turns critical.

2. Correct Check Definitions

If the check path, port, or interval is wrong, update the definition and re‑register. For an HTTP check, ensure the endpoint returns 2xx within the timeout. Shorten the interval (e.g., 10s) for faster detection, but avoid intervals under 5s to reduce agent load.

3. Manually Deregister Stale Entries

When a service is already critical and you need immediate cleanup, use the API or CLI (requires service:write on that service):

# Via CLI (run on the node that owns the service)
consul services deregister -id web-1234

# Via API (from any authorized host)
curl -X PUT -H "X-Consul-Token: $TOKEN" \
  "https://consul.example.com/v1/agent/service/deregister/web-1234"

Verify removal with consul catalog service web.

4. Force‑Leave Dead Nodes

If a node is confirmed powered off or destroyed, remove it from the Serf cluster so its services are purged from the catalog. Run on a server (requires node:write):

consul force-leave 

Risk: Force‑leaving a node that is only network‑partitioned but still alive will cause it to rejoin and potentially re‑register stale services. Confirm the node is truly down (e.g., cloud provider shows terminated, no process listening on 8301/8300) before proceeding.

5. Restore Anti‑Entropy Sync

If agents cannot sync with servers, check network connectivity, ACL tokens, and server health. Restart the agent on the affected node after resolving the underlying issue; it will re‑sync and reconcile the catalog.

Escalation Criteria

Stop manual remediation and engage platform engineers when:

• Catalog diverges across servers (different service lists for the same query).
• consul operator raft list-peers shows a leader but followers with stale indexes or "failed" health.
• Autopilot reports unhealthy servers (consul operator autopilot get-config and get-health).
• Services reappear after successful deregistration, indicating a server‑side registration loop or misconfigured external catalog sync.

In these cases, investigate Raft stability, snapshot integrity, and any automation (e.g., Consul‑Terraform‑Sync, Kubernetes sync) that might be re‑registering services.

Practical Verification

To confirm the fix works in your environment:

1. Register a test service with a short HTTP check and DeregisterCriticalServiceAfter: "10s".
2. Kill the backing process (e.g., pkill -9 test-server).
3. Poll /v1/health/service/test?passing=false every 2 seconds and note when the entry disappears.
4. Repeat without the deregistration field and observe the entry persisting until the check TTL expires (or indefinitely for script/TTL checks).

Compare consul catalog services output before and after remediation to ensure stale entries are gone.

Limitations and Caveats

• Version differences: Field names (e.g., deregister_critical_service_after vs. DeregisterCriticalServiceAfter) and default intervals vary across Consul versions. Always verify against the documentation for your deployed version (this guide assumes 1.12+).
• Partition tolerance: Setting DeregisterCriticalServiceAfter too low (e.g., <5s) can cause healthy instances to be deregistered during brief network blips. Choose a value that exceeds your expected partition duration.
• Force‑leave safety: Only force‑leave nodes you have confirmed are permanently gone. In cloud environments, use instance metadata or provider APIs to verify termination.
• ACL tokens: All CLI and API commands shown require appropriate ACL policies. Operations on the catalog need service:write; node operations need node:write.

Rollback Considerations

Manual deregistration and force‑leave are destructive operations that change cluster state. There is no automated rollback; if you deregister the wrong service ID, you must re‑register it. Always capture the service definition (consul catalog service -format=json) before deletion.