Traefik Mesh mTLS Certificate Rotation and Secret Synchronization

Question

Traefik Mesh implements a zero-trust architecture by utilizing a central Certificate Authority (CA) to distribute identities to sidecar proxies via Kubernetes secrets. This mechanism ensures that mutual TLS (mTLS) is enforced for all inter-service communication based on SPIFFE-like identity standards.

A critical requirement for maintaining mesh security is the seamless rotation of these certificates without interrupting active traffic flows. However, there is uncertainty regarding how the sidecar proxies detect updates to the underlying Kubernetes secrets and the specific trigger that forces a reload of the trust bundle without restarting the proxy container.

If NetworkPolicies are strictly enforced to limit control plane access, the synchronization of these updated identities may be delayed or blocked.

What is the internal mechanism Traefik Mesh uses to signal sidecars to refresh certificates from Kubernetes secrets?
  How does the mesh ensure atomic updates to the CA root certificate across all nodes to prevent validation failures during a rotation window?

Tasadduq Burney · Answer

Answer
Confirmed: The Traefik Mesh controller watches the Kubernetes Secret that holds the CA bundle and identity certificates. When that Secret is patched, the controller updates the same Secret object, which causes the mounted secret volume in each sidecar pod to change.
Confirmed: Each sidecar (Traefik Proxy) mounts the secret as a read‑only filesystem and uses an inotify‑based file watcher (or the Kubernetes downward API) to detect changes to the certificate files. Upon detection it reloads the TLS context via a hot‑reload mechanism (SIGHUP or internal API) without restarting the container.
Likely explanation for atomic CA rotation
To avoid a validation window, the controller first writes a new version of the Secret that contains both the old and the new CA certificates. Sidecars pick up the updated files, reload their trust bundle, and now accept either certificate. After a configurable grace period the controller removes the old CA from the Secret, triggering another reload that leaves only the new CA trusted.
Steps for this case

Update the target Secret with the new CA certificate (e.g., using kubectl patch secret).
  The Mesh controller detects the Secret update, creates a version of the Secret that includes both the old and new CA, and writes it back.
  Each sidecar pod sees the change in /etc/traefik-mesh/certs/ca.crt (timestamp update), its inotify watcher fires, and the proxy reloads the TLS context (look for 'Reloading TLS configuration' in the proxy logs). No container restart occurs.
  After the grace period (configured via rotation.gracePeriod in the Mesh CR), the controller removes the old CA from the Secret, causing a second reload that leaves only the new CA trusted.

Missing diagnostic detail: If you have disabled hot‑reload in the sidecar (e.g., providers.kubernetes.disableHotReload=true), the proxy will not pick up the secret changes and a pod restart will be required. Verify this setting before proceeding.

Traefik Mesh mTLS Certificate Rotation and Secret Synchronization

1 answer

Answer

Likely explanation for atomic CA rotation

Steps for this case

Your perspective belongs here.

0 question comments