Consul Connect mTLS and Health Checks for Internal Service Trust

Problem and takeaway

You run several internal services on shared hosts or virtual machines, and any service can reach any other by IP because the network assumes everything inside the VPC is trustworthy. A compromised or misconfigured service can scan and call internal APIs. The useful takeaway is to run a small Consul cluster, register services with health checks, and enforce service-to-service authorization with Consul Connect intentions and mTLS sidecars so trust follows service identity rather than network location.

Requirements

• Internal services must discover each other without hardcoded IP addresses.
• Unhealthy instances must be removed from discovery automatically.
• Service-to-service traffic must be authenticated and authorized by identity, not by subnet.
• The design must stay small enough to operate with a few servers and client agents.

Smallest suitable design

Run three Consul server nodes for Raft quorum. Raft is the consensus protocol Consul servers use to replicate state; three servers tolerate one failure while still accepting writes. Run a Consul client agent on every host that runs services. Clients forward DNS and API requests to the servers and host local Connect sidecar proxies.

Register each service with an HTTP health check. When a check fails repeatedly, Consul deregisters the service from DNS and catalog results so callers stop sending traffic to it. Deploy a Connect sidecar proxy next to each service instance. The sidecar terminates mTLS for inbound connections and initiates mTLS for outbound connections. Consul intentions are named authorization rules that state whether one service may connect to another.

Example configuration

Run this on each Consul server node. Replace placeholders with your values.

{
  "server": true,
  "bootstrap_expect": 3,
  "datacenter": "{{datacenter}}",
  "data_dir": "/opt/consul/data",
  "bind_addr": "0.0.0.0",
  "client_addr": "0.0.0.0",
  "ports": {
    "dns": 8600,
    "http": 8500
  },
  "encrypt": "{{base64-serf-key}}"
}

Permissions: root or the consul user, depending on how the service is installed. Risk: binding client_addr to 0.0.0.0 exposes the HTTP API on all interfaces; restrict it with a firewall or bind to the private interface only.

Run this on each client host.

{
  "datacenter": "{{datacenter}}",
  "data_dir": "/opt/consul/data",
  "bind_addr": "0.0.0.0",
  "client_addr": "127.0.0.1",
  "retry_join": [
    "{{consul-server-1}}",
    "{{consul-server-2}}",
    "{{consul-server-3}}"
  ],
  "ports": {
    "dns": 8600,
    "http": 8500
  }
}

Register a service with a sidecar and health check using a configuration file or the HTTP API.

{
  "service": {
    "name": "api",
    "port": 8080,
    "connect": {
      "sidecar_service": {}
    },
    "check": {
      "http": "http://127.0.0.1:8080/health",
      "interval": "10s",
      "timeout": "2s",
      "deregister_critical_service_after": "30s"
    }
  }
}

Create a deny intention between two services. Run on a host with Consul CLI access and an ACL token that can write intentions.

consul intention create -deny web api

Expected check: traffic from the web sidecar to the api sidecar fails, while traffic allowed by an existing intention succeeds. Rollback: run consul intention delete web api to remove the rule. Under a default-deny policy this immediately blocks all traffic between the services, so verify the dependency map before deleting in production.

Trust and data boundaries

Boundary	Protection	Limitation
Consul gossip LAN between servers and clients	Encrypted with Serf keys	Protects control-plane membership traffic, not application payloads
Server RPC WAN layer	Server-to-server RPC communication	Cross-datacenter traffic needs separate WAN encryption planning
Connect data plane between sidecars	mTLS with certificates issued by Consul CA or external PKI	Intentions are enforced at the proxy; direct IP connections bypass them
Sidecar to local application	Localhost trust boundary	Plaintext by default; compromise of the host exposes this segment

Consul DNS runs on port 8600 and returns SRV records for healthy services. The HTTP API runs on port 8500 and handles registration, intentions, and status queries. DNS does not return health status directly; callers must handle SRV record changes and retry when instances disappear.

Operational checks

• Raft leader stability: query http://{{consul-server-ip}}:8500/v1/status/leader on each server. An empty response means no leader, which blocks writes such as service registration and intention changes.
• Server peer count: query /v1/status/peers and confirm the expected number of voting servers.
• Disk usage: monitor the data_dir path; snapshots and Raft logs grow with write volume.
• Client membership: run consul members on a client or server and alert on unexpected leave or join events.
• Connect certificate rotation: inspect sidecar proxy logs for certificate renewal errors; Consul Connect issues short-lived certificates.
• ACL enforcement: call the HTTP API with and without a token and confirm the expected access difference. Default ACL behavior changed across Consul versions, so verify whether ACLs are enabled on your installed version.

Failure modes

• Loss of quorum: with three servers, losing two prevents writes. Reads may continue from cached client state, but new registrations and intention updates fail until quorum returns.
• Network partition: clients isolated from servers may mark services unhealthy after the check threshold, removing them from discovery even if the application is healthy.
• Misconfigured intentions: a default-deny rule or a missing allow rule breaks legitimate service traffic immediately.
• Direct IP bypass: applications that connect to a pod or VM IP instead of the local sidecar are not subject to intentions.
• Script health checks: slow or frequent scripts consume server resources and add latency across many services.
• Raft write pressure: service registration and intention updates are linearized through the leader; high write volume increases latency.

Verification

1. Run on a client host: dig @127.0.0.1 -p 8600 {{service-name}}.service.consul SRV. Confirm healthy instances appear.
2. Fail the service health check temporarily and repeat the DNS query. Confirm unhealthy instances disappear after the deregistration threshold.
3. Test an allowed intention: make a request through the source sidecar to the destination sidecar and confirm success.
4. Test a denied intention: make the same request and confirm connection failure.
5. Inspect sidecar logs and certificate SANs to confirm mTLS is active.

Conditions that would change the design

• Cross-datacenter deployment: add WAN federation, separate datacenter configuration, and review gossip encryption across the WAN.
• External certificate authority: replace Consul CA with Vault or another external PKI for Connect certificates.
• Ingress gateway model: move from sidecar-to-sidecar trust to a gateway termination pattern for north-south traffic.
• Strict ACL requirements: enable ACLs with tokens and bind the HTTP API to private interfaces only.

Limitations

Consul does not encrypt application payloads by default outside Connect mTLS. Plaintext traffic between the sidecar and the local application remains inside the host trust boundary. Consul DNS does not indicate health, so clients must handle empty SRV responses and connection retries. This design assumes a single datacenter and internal service-to-service traffic; it does not replace perimeter controls for external clients.