Diagnosing and Fixing TraefikMesh mTLS Certificate Rotation Failures

Recognizable Symptoms

When a TraefikMesh deployment starts to misbehave after 24‑48 hours, you’ll typically see one or more of the following:

• HTTP 502/503 responses from services that were previously healthy.
• Sidecar proxy logs containing certificate expired or certificate verify failed messages.
• Controller pod logs with failed to generate server certificate or rotation error.

These symptoms usually indicate that the mTLS certificates used by sidecar proxies have become invalid or the controller cannot renew them.

Root Cause Table

Root Cause	Typical Indicator	Why It Happens
Clock skew between controller and sidecar nodes	Controller logs show rotation interval exceeded	The controller’s rotation timer is based on its system clock; if it is ahead of the nodes, certificates appear expired.
Mis‑configured cert‑manager integration	Sidecars never receive a new traefik-mesh-ca secret	The ClusterIssuer is not pointing to the secret that the controller creates.
Webhook/sidecar injection failure	Pods continue to use old certificates after rotation	The mutating webhook is down or RBAC prevents it from adding the sidecar.
NetworkPolicy blocking controller ports	Health checks to traefik-mesh-controller:8443 time out	Policy blocks TCP 8443 or 8080 between namespaces.

Ordered Verification Steps

1. Verify controller and sidecar pods are running

   kubectl get pods -n traefik-mesh
   # Expect all controller and sidecar pods to be in the Running state.
   

2. Check controller logs for rotation activity

   kubectl logs -n traefik-mesh -l app=traefik-mesh-controller
   # Look for lines containing "rotation" or "certificate".
   

3. Inspect sidecar certificates

   kubectl exec -n  -c traefik-mesh <POD_NAME> -- \
     openssl x509 -noout -dates
   # Compare the notAfter value to the current time.
   

4. Confirm webhook availability

   kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
     | grep traefik-mesh
   # Ensure the mutating webhook has a valid service reference.
   

5. Verify RBAC for certificate signing

   kubectl get clusterrolebinding -o yaml | grep -A5 traefik-mesh
   # Ensure the binding grants the controller permission to sign CSRs.
   

Targeted Fixes

1. Resolve Clock Skew

All cluster nodes should run a reliable NTP or Chrony service.

# On each node (requires sudo)
chronyc -a makestep
# Restart the controller pod to force a new rotation cycle.
kubectl rollout restart deployment traefik-mesh-controller -n traefik-mesh

2. Correct cert‑manager Integration

Ensure the ClusterIssuer references the CA secret created by TraefikMesh.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: traefik-mesh-issuer
spec:
  ca:
    secretName: traefik-mesh-ca
    namespace: traefik-mesh

After applying the issuer, annotate the controller to trigger a temporary certificate request.

kubectl annotate pod -n traefik-mesh traefik-mesh-controller \
  cert-manager.io/issue-temporary-certificate=true

3. Force Sidecar Re‑Injection

When the webhook is healthy but sidecars still hold old certs, restart the affected deployments or add the injection annotation.

# Restart deployment
kubectl rollout restart deployment -n  
# Or annotate the pod template
kubectl patch deployment -n   \
  --type merge -p '{"spec":{"template":{"metadata":{"annotations":{"traefik-mesh.io/inject":"true"}}}}}'

4. Open Controller Ports in NetworkPolicy

Allow TCP 8443 (control plane) and 8080 (metrics) from mesh namespaces.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-traefik-mesh
  namespace: 
spec:
  podSelector: {}
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: traefik-mesh
    ports:
    - protocol: TCP
      port: 8443
  - to:
    - namespaceSelector:
        matchLabels:
          name: traefik-mesh
    ports:
    - protocol: TCP
      port: 8080
  policyTypes:
  - Egress

Escalation Criteria

• If certificate rotation fails after two consecutive intervals despite applying all fixes, suspect the CA private key has been corrupted. Re‑install the entire TraefikMesh stack.
• Controller pods enter CrashLoopBackOff with errors like failed to generate server certificate; this indicates a deeper issue in the signing process and requires a full reinstall or migration.
• If more than 10 % of sidecars report expired certificates simultaneously, the controller may be unreachable. Engage Traefik Enterprise support (if applicable) or plan a migration to a supported service mesh.

Limitations & Best Practices

• TraefikMesh is archived (last release 1.4.0) and no longer receives security patches. Consider migrating to Traefik Enterprise or a community mesh like Istio.
• mTLS is mandatory; it cannot be disabled on a per‑service basis. Use traefik-mesh.io/opt-out: "true" to bypass the mesh for specific workloads.
• Certificate rotation relies on the Kubernetes CSR API; ensure the cluster’s signerName is functional.
• Always test changes in a staging namespace before applying to production. Verify that sidecar certificates renew by running openssl x509 -checkend 86400 after rotation.