Consul Health Check Failure Troubleshooting Guide

When a Consul‑registered service shows a critical status even though its process is running, the issue is almost always a connectivity or configuration problem. This guide walks through the most common root causes, shows how to verify each, and lists concrete fixes and escalation steps.

Problem Statement

A Consul service is registered but the UI/API reports its status as critical or warning despite the application process running. The goal is to isolate the cause, apply the appropriate fix, and confirm the service returns to passing status.

Quick Cause‑Check Table

Cause	Typical Symptoms	Primary Check
Network partition or firewall blocking the check port	Consul logs show connection refused or timeout	Ping/port test from agent node
Application timeout – check fails to respond in time	Check logs show `check failed` after interval	Reduce interval or increase timeout
Incorrect health‑check definition (wrong path/port)	Check never succeeds; agent logs show 404 or unreachable	Validate JSON registration
Resource exhaustion causing the service to hang	High CPU/Memory, check stalls, then fails	Monitor node resources

Ordered Diagnostic Checklist

Verify Service Registration
Run on the Consul agent node:
```
consul catalog services | grep <service-name>
```
Expected output: the service name appears in the list. If missing, registration failed.
Check Health‑Check Endpoint Connectivity
From the same node, attempt to reach the check endpoint:
```
# HTTP check
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:<port>/<path>
# TCP/TTL check – simple port test
nc -zv 127.0.0.1 <port>
```
Success: HTTP 200 or TCP open. Failure: connection refused, timeout, or non‑200 status.
Inspect Consul Agent Logs
Search for check‑related entries:
```
journalctl -u consul | grep -i check | tail -n 20
```
Look for messages like check failed or connection refused.

Validate Registration JSON

If you registered the service via a JSON file, review its contents. Example:

{
  "ID": "web-1",
  "Name": "web",
  "Address": "10.0.0.5",
  "Port": 8080,
  "Check": {
    "HTTP": "http://10.0.0.5:8080/health",
    "Interval": "10s",
    "Timeout": "1s",
    "DeregisterCriticalServiceAfter": "1m"
  }
}

Common mistakes: wrong host, wrong port, wrong path, missing quotes, or an illegal field name.

Monitor Node Resources
Check CPU and memory on the agent node:
```
top -bn1 | grep -i cpu
free -m
```
High load or swap usage can delay health checks.
Test with a Minimal Check
Temporarily replace the original check with a simple script that always returns success:
```
{
  "ID": "web-1",
  "Name": "web",
  "Check": {
    "Script": "echo 0",
    "Interval": "5s"
  }
}
```
If the service becomes passing, the original check logic is at fault.

Fixes Tied to Findings

Network/Firewall Issues
Ensure the Consul agent can reach the check port from the same host. Open or allow the port in the firewall, or adjust the check endpoint to use a reachable host/IP.
Timeout/Interval Misconfiguration
Increase Timeout or reduce Interval so the check has enough time to succeed. Avoid setting Interval > 30s as it delays failure detection.
Incorrect Check Definition
Correct the HTTP URL, port, or path. Validate the JSON against the Consul service registration schema. After editing, re-register the service with consul services register <file.json> and verify.
Resource Exhaustion
Scale the node, add resources, or move the service to a less loaded host. Consider running the service in a container with enforced limits.
Health‑Check Flapping
If the service repeatedly toggles between passing and critical, increase DeregisterCriticalServiceAfter or add a TTL check that the application updates on a heartbeat.

Escalation Criteria

If after applying all fixes the service remains critical for > 2 intervals, involve the application owner to review code or deployment.
Persistent connectivity failures despite correct firewall rules suggest a deeper network partition – involve network ops.
Repeated flapping after configuration changes may indicate a bug in the service’s health endpoint; trigger a code review or QA test.

Practical Verification Checklist

Run consul catalog services – service listed.
Run consul catalog nodes – node shows healthy status.
Query the service health API: curl http://127.0.0.1:8500/v1/health/service/<service-name> – should return status "passing".
Confirm no check errors in consul logs for the last 5 minutes.

Key Takeaway

Consul health‑check failures are almost always network or configuration problems. A systematic approach—verify registration, test connectivity, review logs, correct JSON, and monitor resources—will quickly bring a critical service back to passing status.

Diagnosing and Fixing Consul Service Health Check Failures