Automatic mTLS Certificate Rotation in Traefik Mesh: Design and Operational Checks

Requirements

Before enabling automatic mTLS in Traefik Mesh, the cluster must meet the following prerequisites:

• Kubernetes ≥ 1.20 with RBAC enabled.
• Traefik Mesh control plane installed (v1.3 or later for external issuer plug‑in support).
• Sidecar injection enabled for the target namespaces (via the traefikmesh.io/sidecar-injection: enabled annotation).
• Network policy that allows traffic on the sidecar’s UNIX domain socket path (/var/run/traefikmesh/certs.sock) only from the same pod.

The goal is to establish zero‑trust service‑to‑service communication without manual certificate handling, while keeping the operational overhead low.

Smallest Suitable Design

The minimal design consists of three components:

1. Control Plane Certificate Controller – watches a TrafficTarget resource, issues short‑lived X.509 certificates (default TTL 24 h) using the built‑in issuer, and pushes the certificate and private key to each sidecar over a UNIX domain socket.
2. Sidecar Proxy (Envoy‑compatible) – loads the certificate from the socket into its TLS context, presents it during mTLS handshake, and trusts the control plane‑issued CA for peer verification.
3. TrafficTarget – defines which services participate in mTLS and whether the mode is strict (drop traffic on validation failure) or permissive (allow fallback to plaintext).

No Kubernetes Secrets are used; certificates reside only in the control plane’s memory and are transmitted via the socket, reducing secret‑management surface.

Trust/Data Boundaries

Trust boundaries are drawn as follows:

• Control Plane → Sidecar: The socket is created with mode 0600 and owned by the sidecar’s UID; only processes inside the same pod can read it. This prevents other pods from stealing the certificate.
• Sidecar → Peer Sidecar: mTLS ensures that both ends present a certificate signed by the control plane’s CA. The CA private key never leaves the control plane.
• Data Plane: Application traffic is encrypted end‑to‑end between sidecars; the application itself sees plaintext after the sidecar decrypts.

Because the control plane does not persist certificates to etcd or Secrets, a compromise of the control plane’s memory is required to obtain a valid cert.

Operational Checks

To verify that automatic rotation works as expected, perform the following steps:

1. Deploy a test application with two services, frontend and backend, each annotated for sidecar injection.
2. Create a TrafficTarget that enables strict mTLS:

apiVersion: traefikmesh.io/v1alpha1
kind: TrafficTarget
metadata:
  name: frontend-to-backend
  namespace: demo
spec:
  destination:
    kind: Service
    name: backend
    namespace: demo
  sources:
    - kind: Service
      name: frontend
      namespace: demo
  # strict mode drops connections if mTLS fails
  trafficPolicy:
    mTLS:
      mode: strict

1. Check the sidecar’s certificate – exec into any frontend pod and read the certificate from the socket:

# Run inside the frontend pod (requires kubectl exec permission)
kubectl exec -it deploy/frontend -n demo -- \
  sh -c "openssl s_client -connect localhost:15001 -showcerts /dev/null \
    | openssl x509 -noout -dates -subject"

The command contacts the sidecar’s admin listener (port 15001 by default) and prints the certificate’s NotBefore, NotAfter, and subject. You should see a validity period close to the configured TTL (e.g., 24 h).

1. Observe rotation – wait longer than the TTL (or temporarily set spec.certificateTTL: 5m in the MeshConfig for faster testing) and repeat the openssl check. The NotAfter timestamp should advance without any pod restart.
2. Log verification – sidecar logs contain lines like:

[info] Received new certificate, TTL 24h
[info] Updating TLS context with new certificate

These messages confirm that the control plane pushed a new cert and the sidecar reloaded its TLS context.

Failure Modes

Understand what happens when parts of the design fail:

• Control Plane Unavailable: Existing sidecars continue to use the current certificate until its NotAfter expires. After expiration, new mTLS handshakes fail, causing connection errors. Existing long‑lived TCP connections (e.g., HTTP/2) may persist until the application layer times out.
• Socket Permissions Misconfigured: If the sidecar cannot read the socket, it will log a permission error and fall back to permissive mode (if enabled) or drop traffic in strict mode.
• CA Key Compromise: An attacker who extracts the control plane’s CA key can forge certificates for any service. Rotating the CA key requires a control plane restart and re‑issuing all service certificates.
• Custom Issuer Not Supported (pre‑v1.3): Attempting to point the controller at Vault or cert‑manager will be ignored; the built‑in issuer will still be used, potentially causing a mismatch with external policies.

Conditions That Would Change the Design

Re‑evaluate the architecture if any of the following become true:

• Operational requirements demand certificate lifetimes shorter than 5 minutes; the control plane’s push interval may need tuning to avoid excessive socket traffic.
• The organization mandates that all cryptographic material be stored in a Kubernetes Secret for auditability; in that case, you would need to patch the controller to write certs to a Secret and adjust the sidecar to read from there.
• You plan to integrate with an external identity provider (e.g., SPIFFE) that issues certificates outside the mesh; this would replace the built‑in issuer and require a custom controller or a mesh‑level certificate‑rotation plugin.
• Network policies restrict UNIX domain socket access to a non‑root user; you would need to run the sidecar as that specific UID/GID and adjust the socket file permissions accordingly.

---

Note: The commands and examples above illustrate the verification process; they have not been executed in a live environment. Adjust resource names, namespaces, and TTL values to match your deployment.