Gardener v1.87+ shoot API server certificate rotation: transient x509 failures when control plane restarts during renewal

Answer

The rotation‑to‑reload window is not formally bounded; it is eventually consistent by design. Gardener publishes the new CA bundle to the shoot status as soon as the serving‑certificate secret is marked Ready, without waiting for all kube‑apiserver pods to reload the new TLS files. Consequently, if a control‑plane pod restarts after the secret update but before its file‑watcher reloads, it can serve a certificate that clients no longer trust, producing transient x509: certificate signed by unknown authority errors. Concurrent autoscaling of the API server during rotation widens this race because new replicas may start with the updated secret while old replicas still serve the old certificate, and the new CA bundle has not yet been propagated to clients.

Confirmed facts (from upstream Kubernetes and Gardener v1.87 behavior)

• Kubernetes kube‑apiserver (≥v1.22) watches the TLS certificate and key files and reloads them automatically; the default refresh interval is 10 minutes (--tls-cert-file-refresh-interval) and is not exposed in the Shoot spec.
• The Gardener certificate‑rotation controller updates the serving‑certificate secret via cert‑manager, then immediately updates the shoot status credentials (caBundle, kubeconfig) once the secret’s Ready condition is true. There is no built‑in barrier that waits for all API‑server pods to confirm reload.
• In practice the reload typically occurs within one refresh interval (≤10 min) but can be longer under I/O pressure or if the file‑watcher misses the event.
• Gardener v1.87 does not provide a synchronization mechanism (e.g., waiting for a reload‑ready lease or pod annotation) – this gap has been discussed in SIG meetings and tracked in issues such as gardener/gardener#7000‑series.
• Cluster autoscaler or Gardener‑controlled replica changes during rotation create additional windows where new pods read the updated secret while the old CA bundle is still present in clients, and vice‑versa.

Steps needed for this case

1. Trigger rotation during a low‑traffic window and annotate the shoot:

   kubectl annotate shoot <shoot-name> gardener.cloud/operation=rotate-certificates --overwrite

2. Watch the serving‑certificate secret in the seed namespace:

   kubectl get secret -n <seed-namespace> <tls-serving-secret> -o yaml -w

3. When the secret shows Ready: True, monitor the shoot status credentials for the new CA bundle:

   kubectl get shoot <shoot-name> -o jsonpath='{.status.credentials.kubeconfig}' | base64 -d | grep certificate-authority-data

4. After the status change, verify that all kube‑apiserver pods have reloaded the new TLS files by checking their logs for a “TLS certificate reloaded” message (log level 4+):

   kubectl logs -n kube-system -l component=kube-apiserver | grep -i "TLS certificate reloaded"

5. If you observe x509 errors from clients, wait until the log confirms reload on every replica, then retry the client connection.
6. (Optional) If you have a custom --tls-cert-file-refresh-interval set via spec.controlPlaneConfig.kubeApiserver.extraArgs, note that value; otherwise assume the default 10 min and plan accordingly.

Missing diagnostic detail that would change the recommendation

Do you have a custom tls-cert-file-refresh-interval configured for the shoot’s kube‑apiserver (via controlPlaneConfig)? Knowing the actual interval determines how long you must wait after the secret update before expecting all API‑server pods to have reloaded.