Troubleshooting Consul Connect mTLS Handshake Failures

The Problem: mTLS Handshake Failures

When deploying a service mesh with Consul Connect, you may encounter a scenario where services cannot communicate despite being correctly registered. The primary symptom is a failed to handshake or certificate verify failed error in the sidecar proxy logs, while the service health checks remain critical or report connectivity timeouts.

The takeaway: Most mTLS failures in Consul Connect are caused by missing Service Intentions (the mesh’s firewall) or a mismatch between the sidecar’s expected upstream configuration and the actual service registration.

Diagnostic Matrix

Symptom	Likely Cause	Primary Diagnostic Tool
"certificate verify failed" in logs	Expired or untrusted CA certificates	consul tls ca file
Connection timeout / No response	Deny Intention in place	consul intention list
"no such host" or "connection refused"	Incorrect proxy upstream config	consul connect proxy -config
Proxy fails to start/initialize	Consul version < 1.8 or missing -connect	consul version

Step‑by‑Step Connectivity Checks

Perform these checks in order to isolate the failure point from the infrastructure level up to the application layer.

1. Agent Health and Versioning

Ensure all agents are running and support the Connect feature. Consul Connect requires version 1.8 or later.

# Run on the node hosting the service
consul members
consul version

Check: Verify that all nodes are listed as alive. If a node is failed or left, the sidecar cannot retrieve the necessary TLS certificates from the local agent.

2. Service Registration Verification

A service must be explicitly registered with Connect enabled to receive a sidecar proxy and TLS certificates.

Check: Inspect the service definition. If using a configuration file, ensure the connect { sidecar_service = true } block exists. If using the CLI, verify the -connect flag was used during registration.

3. Intentions Validation

Consul Connect uses a deny‑by‑default security model. If no explicit allow intention exists between the source and destination services, the proxy will terminate the connection during the handshake.

# Run with an ACL token possessing service:read permissions
consul intention list

Check: Look for the specific source and destination pair. If the status is deny or is missing entirely, the connection will fail.

4. Proxy Configuration and Trust Store

Verify that the sidecar knows where to route traffic and trusts the root Certificate Authority (CA).

# Check the CA certificate path and validity
consul tls ca file

Check: Ensure the CA certificate is present and has not expired. If the proxy logs indicate an unknown CA, the trust store is likely out of sync with the Consul server’s CA.

Remediation Steps

• For Denied Intentions: Create an allow intention. Note that this requires an ACL token with node:write and service:read permissions.

  # Run on a Consul CLI client
  consul intention create -deny=false -source-name=web-service -destination-name=api-service
  

  Note: Intention changes are eventually consistent; allow up to 30 seconds for propagation.
• For CA Failures: Renew the Consul PKI and redistribute the new CA certificate to all agents. Restart the sidecar proxies to pick up the new trust bundle.
• For Proxy Config Errors: Correct the local service address or upstream definitions in the service registration and reload the sidecar proxy.

Verification of Fix

To confirm the resolution, execute a request from within the sidecar container to the local proxy port:

# Run inside the sidecar container
curl -v https://localhost:<proxy_port>

Expected Result: A 200 OK response. If you see SSL_ERROR_SYSCALL or certificate expired, the mTLS handshake is still failing.

Escalation Criteria

• Security/PKI Team: Escalate if unknown CA errors persist after a full CA renewal and agent restart.
• Network/Infra Team: Escalate if intentions are set to allow but packet captures show traffic being dropped before reaching the proxy.
• HashiCorp Support: Escalate if the proxy crashes repeatedly with segmentation faults or internal errors despite correct configuration.