Question
Consul Connect and Envoy Proxy mTLS Interoperability After Certificate Rotation
Nia Byte
0 reputation · 19 Apr 2021, 07:18 UTC
36.5K views1
Loading…
0 reputation · 19 Apr 2021, 07:18 UTC
Consul Connect automatically manages mTLS certificates through its internal CA, distributing rotated certificates to Envoy sidecars for service-to-service encryption. The certificate rotation process involves Consul generating new leaf certificates and updating the Connect CA bundle, which Envoy proxies must then reload to maintain encrypted communication.
However, the timing and mechanism of certificate propagation between Consul agents and Envoy sidecars varies across Consul versions. In some configurations, Envoy may cache certificates beyond their validity period, causing connection failures when applications attempt to communicate with newly rotated certificates. The Connect intention system continues to enforce authorization policies during rotation, but mismatched certificate states between proxy and server can create transient communication gaps.
The integration boundary between Consul's certificate lifecycle management and Envoy's secret discovery service (SDS) implementation requires careful coordination. Applications relying on long-lived connections may experience unexpected disruptions if the rotation timing is not properly synchronized with connection lifecycle management.
Which Consul and Envoy version combinations ensure seamless certificate rotation without service disruption, and how can the SDS update timing be verified to prevent connection failures during the rotation window?
35775 reputation · 19 Apr 2021, 09:36 UTC
There is no single “one‑size‑fits‑all” version pair that guarantees zero‑downtime rotation in every environment. The proven combinations are the Envoy version that ships with or is officially tested against a given Consul release. For example, Consul 1.13.x is bundled with Envoy 1.25.x, and Consul 1.14.x ships with Envoy 1.26.x. Using those pairs, the Consul agent updates the leaf and CA secrets, Envoy’s SDS client pulls them, and the proxy reloads the secrets before the current certificates expire.
consul
envoy# Consul version
consul --version
# Envoy version (bundled with Consul)
envoy -v
# In the Consul UI or via API
curl http://localhost:8500/v1/agent/health/service/your-service
# Look for "Certificate Not Before/After" timestamps
# In envoy.yaml
admin:
access_log_path: /dev/stdout
address:
socket_address:
address: 127.0.0.1
port_value: 8001
# Then view SDS metrics:
curl http://127.0.0.1:8001/stats?format=json | jq '.stats[] | select(.name | contains("sds.") )'
# Compare Consul leaf expiry with Envoy’s current leaf expiry
# Fetch Envoy’s active leaf via admin API
curl http://127.0.0.1:8001/debug/secret?name=consul-leaf | jq '.leaf_certificate.not_after'
# Compare with the timestamp in Consul
# Deploy a short‑lived client
# Generate traffic after rotation
# Observe that new handshakes succeed while old connections drain
To fine‑tune the recommendation, please provide:
consul-connect-envoy binary (bundled) or a separate Envoy instance consuming Consul SDS.tls_certificate_reuse or custom SDS configuration.Use comments to ask for clarification. Post a solution as an answer.
35,775 reputation · 19 Apr 2021, 17:33 UTC
The existing answer correctly identifies version pairing as necessary but not sufficient. A critical factor is whether Envoy consumes certificates via SDS (Secret Discovery Service) or file-based mounts. Only SDS provides atomic, in-memory secret updates without proxy reloads; file-based rotation requires a SIGHUP or restart, which introduces a window where the proxy serves stale leaf certificates.
connect.hashicorp.com/proxy-sidecar-sds-enabled: "true".grace_period (e.g., 10m) in the Connect CA configuration so Consul publishes the new leaf before the old one expires.consul connect ca leaf rotated log lines with Envoy's sds.config_update counter to prove the new secret arrives before not_after.Without SDS and an adequate grace period, even matched version pairs can exhibit transient handshake failures.