Choosing a Certificate Authority for Consul Connect mTLS: Built-in, Vault, or External CA

The Decision

Consul Connect requires a certificate authority (CA) to issue leaf certificates for service-to-service mutual TLS. The choice of CA provider affects operational complexity, security posture, and scalability. This guide compares the three supported modes—built-in Consul CA, HashiCorp Vault PKI, and an external private CA—so you can select the option that matches your constraints.

Constraints to Consider

• Trust scope: Single datacenter or multi-datacenter federation.
• Compliance: Need for hardware security modules (HSM), audit logs, or root-key lifecycle control.
• Operational capacity: Ability to run and monitor an additional Vault cluster or a manual signing pipeline.
• Performance sensitivity: Latency tolerance for certificate issuance (sub-millisecond vs. network round-trip).
• Consul edition: Enterprise features like namespace isolation and partition-level CA are unavailable in Community edition.

Supported Options at a Glance

Trade-offs

Built-in Consul CA

Simplest to start: no external services, automatic root rotation, and local issuance. The root private key resides on Consul server disks; compromise requires a full mesh re-key. Encrypt storage volumes and restrict server access. Suitable for single-datacenter or simple multi-datacenter deployments where you can replicate the root across DCs.

Vault PKI

Centralizes PKI for Consul and non-Consul workloads. Provides HSM integration, detailed audit logs, and role-based issuance. Requires a highly available Vault cluster and network connectivity from Consul servers to Vault. Consul servers must authenticate to Vault (token, AppRole, or Kubernetes auth); token rotation must be automated to avoid issuance outages. Cross-datacenter trust can leverage Vault replication.

External Private CA

Gives full control over root lifecycle and hardware roots, meeting strict compliance requirements. However, Consul generates CSRs that an operator must sign externally and return via API. No automatic leaf renewal; you must build a renewal pipeline with monitoring on consul.connect.ca.cert.expiry metrics. Operational burden is high, and any signing pipeline failure risks mesh-wide outage when leaf certificates expire.

Concrete Implementation: Vault PKI Integration

Below is a minimal configuration to switch a Consul 1.15+ server cluster from the built-in CA to Vault PKI. Assumes a Vault 1.13+ cluster with the PKI secrets engine enabled at pki/ and a role named consul-connect that issues certificates with appropriate SANs.

1. Prepare Vault

# Enable PKI engine (if not already)
vault secrets enable -path=pki pki

# Create a role for Consul Connect leaf certificates
vault write pki/roles/consul-connect \
    allowed_domains="service.consul" \
    allow_subdomains=true \
    max_ttl="72h" \
    generate_lease=true

2. Configure Consul Servers

On each Consul server, update the agent configuration (HCL or JSON) and reload:

connect {
  enabled = true
  ca_provider = "vault"
  vault {
    address = "https://vault.example.com:8200"
    token   = "${VAULT_TOKEN}"  # Use a periodically rotated token or AppRole
    root_pki_path = "pki/"
    role          = "consul-connect"
  }
}

Restart or reload the Consul server process. The server will now forward CSRs to Vault.

3. Validate the Switch

# Verify CA provider configuration
consul connect ca get-config
# Expected output includes Provider: vault and Vault role details

Register a test service and deploy a sidecar to confirm mTLS handshake:

# On a client node
consul connect envoy -sidecar-for web -admin-bind 127.0.0.1:19000 &
# In another terminal, test upstream connectivity
curl -v --cert /path/to/leaf.crt --key /path/to/leaf.key https://upstream.service.consul

Check that the handshake uses TLS 1.3 and the certificate CN/SAN matches the expected service identity.

Validation Checklist for Any Provider

1. Run consul connect ca get-config to confirm active provider and root expiry.
2. Deploy test sidecars and verify successful mTLS handshake with curl or openssl s_client.
3. Simulate root rotation (built-in only): consul connect ca root rotate and confirm propagation via consul members -wan.
4. Monitor certificate expiry metrics: scrape consul_connect_ca_cert_expiry_seconds via Prometheus; alert at 70% of TTL.
5. Chaos test: stop Vault (if using Vault PKI) or revoke built-in root key; verify Consul logs show issuance failures while existing connections persist until leaf expiry.

Limitations and Gotchas

• Cross-datacenter federation requires all DCs to share the same trust root. Mixing CA providers across DCs is unsupported.
• Leaf certificate TTL must exceed Consul's connect.ca_leaf_ttl and Envoy's rotation grace period. Too short causes rotation storms; too long increases blast radius of compromise.
• External CA mode disables automatic leaf rotation; you must implement a renewal pipeline with alerting.
• Vault token expiration halts certificate issuance. Automate token rotation (e.g., via Vault Agent or CI/CD).
• Consul Enterprise features (namespace isolation, partition-level CA) are not available in Community edition; verify edition before designing multi-tenant mesh.

Practical Verification

After any CA change, run the validation checklist above in a staging environment that mirrors production topology. Use consul connect ca get-config to capture the root expiry timestamp and set a calendar reminder for rotation well before expiry. For Vault PKI, ensure Vault's own seal/unseal procedures are documented and tested.