x509: certificate signed by unknown authority in cattle-cluster-agent

Short answer

The agent validates the Rancher endpoint against exactly one trust source: the CA bundle in the cattle-system/cacerts secret, or the node OS trust store when agent-tls-mode is set to system-store. The x509: certificate signed by unknown authority error means the chain actually served on port 443 is not signed by a CA in that source. When TLS terminates at an external load balancer or proxy, the agent sees the load balancer's certificate rather than Rancher's, and that is the most common cause of this failure.

Confirmed behavior vs. environment-specific detail

Well-established: the agent does not dynamically learn a new chain. It validates against its configured bundle and fails the handshake immediately — there is no fallback to an untrusted chain. Rancher also injects a CA checksum into the agent deployment, so changing the cacerts secret changes that checksum and triggers a rolling restart of the agent pods.

Needs confirmation in your setup: whether the endpoint is re-encrypting to Rancher's own certificate or passing the external certificate straight through, and whether your Rancher version supports agent-tls-mode. That setting is version-sensitive and was introduced in later Rancher 2.x releases; older installations rely on the cacerts secret alone. Verify against your exact version before relying on it.

Steps for this case

1. Read the agent log to capture the exact hostname or IP it tried to reach. That identifies which endpoint and chain are involved.

   kubectl -n cattle-system logs deploy/cattle-cluster-agent --tail=50

2. Inspect the chain the endpoint actually serves and note its issuer.

   openssl s_client -connect rancher.example.com:443 \
     -servername rancher.example.com -showcerts </dev/null 2>/dev/null \
     | openssl x509 -noout -subject -issuer

3. Decode the bundle the agent trusts and compare issuers.

   kubectl -n cattle-system get secret cacerts \
     -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer

4. Choose the matching fix:
   • Add the CA that signed the served certificate to cattle-system/cacerts (or supply it through your Helm values, for example privateCA=true with matching bundle content).
   • Or configure re-encryption so Rancher presents its own certificate to the agent, keeping the generated CA authoritative.
   • Or, where supported, set agent-tls-mode=system-store and install that CA in the node OS trust store.
5. Apply the change, then let the checksum-driven rollout proceed or restart explicitly, and confirm a successful connection in the logs.

   kubectl -n cattle-system rollout restart deploy/cattle-cluster-agent

6. If the CA changed, regenerate the registration command so new clusters receive the current checksum.

Rotation and rolling updates

A rolling update only resolves the error if the updated bundle matches the chain the server genuinely presents. Expect brief agent disconnects while pods restart; avoid scheduling unrelated control-plane maintenance at the same time. Note that cattle-node-agent can have separate trust behavior — check it too if node-level operations fail.

What not to do

Do not disable TLS verification or set insecure-skip-tls-verify as a permanent fix. It weakens security and hides the underlying trust-chain mismatch rather than correcting it.

The one detail that changes the recommendation

Tell me what chain is served on port 443 and whether re-encryption is configured. If the external certificate is passed through, the fix is adding that CA or switching trust mode. If Rancher re-encrypts, the problem is usually a stale or mismatched cacerts bundle, and the fix is updating the secret instead.