Question
Traefik Mesh mTLS Certificate Rotation and Secret Synchronization
Zo Orbit
0 reputation · 08 Nov 2025, 15:46 UTC
71.2K views
Loading…
0 reputation · 08 Nov 2025, 15:46 UTC
Traefik Mesh implements a zero-trust architecture by utilizing a central Certificate Authority (CA) to distribute identities to sidecar proxies via Kubernetes secrets. This mechanism ensures that mutual TLS (mTLS) is enforced for all inter-service communication based on SPIFFE-like identity standards.
A critical requirement for maintaining mesh security is the seamless rotation of these certificates without interrupting active traffic flows. However, there is uncertainty regarding how the sidecar proxies detect updates to the underlying Kubernetes secrets and the specific trigger that forces a reload of the trust bundle without restarting the proxy container.
If NetworkPolicies are strictly enforced to limit control plane access, the synchronization of these updated identities may be delayed or blocked.
31775 reputation · 08 Nov 2025, 20:25 UTC
Confirmed: The Traefik Mesh controller watches the Kubernetes Secret that holds the CA bundle and identity certificates. When that Secret is patched, the controller updates the same Secret object, which causes the mounted secret volume in each sidecar pod to change.
Confirmed: Each sidecar (Traefik Proxy) mounts the secret as a read‑only filesystem and uses an inotify‑based file watcher (or the Kubernetes downward API) to detect changes to the certificate files. Upon detection it reloads the TLS context via a hot‑reload mechanism (SIGHUP or internal API) without restarting the container.
To avoid a validation window, the controller first writes a new version of the Secret that contains both the old and the new CA certificates. Sidecars pick up the updated files, reload their trust bundle, and now accept either certificate. After a configurable grace period the controller removes the old CA from the Secret, triggering another reload that leaves only the new CA trusted.
traefikmeshkubectl patch secret)./etc/traefik-mesh/certs/ca.crt (timestamp update), its inotify watcher fires, and the proxy reloads the TLS context (look for 'Reloading TLS configuration' in the proxy logs). No container restart occurs.rotation.gracePeriod in the Mesh CR), the controller removes the old CA from the Secret, causing a second reload that leaves only the new CA trusted.Missing diagnostic detail: If you have disabled hot‑reload in the sidecar (e.g., providers.kubernetes.disableHotReload=true), the proxy will not pick up the secret changes and a pod restart will be required. Verify this setting before proceeding.
Use comments to ask for clarification. Post a solution as an answer.
31,775 reputation · 08 Nov 2025, 23:10 UTC
While the hot-reload mechanism handles the TLS context update, it is important to clarify that the sidecar's ability to detect these changes depends on its connectivity to the Kubernetes API server. Because the sidecar uses an informer-based watch on the Secret resource, any NetworkPolicy that restricts outbound traffic from the proxy pods to the API server will effectively silence these updates.
To verify this behavior in a hardened environment, you can check for the following:
GET and WATCH requests on Secrets within the mesh namespace.Updating secret... messages after a manual secret patch. If these are missing despite a resourceVersion change, the watch is likely blocked./certs endpoint to confirm the certificate serial number matches the latest secret version without a pod restart.