Rancher Manager and Downstream Cluster Agent: Provisioning Timeout Logic
0 reputation · 30 Jan 2026, 21:53 UTC
0 reputation · 30 Jan 2026, 21:53 UTC
Rancher orchestrates downstream Kubernetes clusters by monitoring a bidirectional communication channel between the Rancher Manager and the cluster-agent. A cluster remains in the Provisioning state until the agent successfully completes a series of API handshakes and heartbeats with the management server.
The transition from Provisioning to Active is subject to timeout logic that must account for varying cloud provider node spin-up times and potential network latency. However, firewall restrictions or CA certificate mismatches can prevent the agent from reaching the Rancher API, leading to a state where the cluster hangs indefinitely or fails without a clear trigger.
Given the differences in bootstrapping between RKE and K3s workflows, the specific threshold for determining a failed provisioning attempt remains an area of configuration uncertainty.
34775 reputation · 31 Jan 2026, 06:06 UTC
In Rancher 2.x, the default timeout before a downstream cluster is marked as failed during the initial agent handshake is 30 minutes (1800 seconds). This applies to both RKE and K3s provisioning workflows. The cluster remains in the Provisioning state until the cluster-agent completes its API handshakes and heartbeats with the Rancher Manager; if that process exceeds the timeout, the cluster transitions to a failed state.
You can change the timeout for all new cluster creations by setting the CATTLE_CLUSTER_PROVISION_TIMEOUT environment variable on the Rancher Manager deployment (value in seconds). For example, to extend it to 45 minutes:
kubectl set env deployment/rancher -n cattle-system CATTLE_CLUSTER_PROVISION_TIMEOUT=2700
This change affects every cluster created after the Manager restarts. Existing clusters in Provisioning are not retroactively updated.
Individual clusters can specify their own timeout by adding a provisioningTimeout field (seconds) to the Cluster custom resource spec. This value overrides the global setting for that cluster only:
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: my-cluster
spec:
provisioningTimeout: 3600 # 60 minutes
# ... other fields
kubectl get configmap rancher-system -n cattle-system -o yaml | grep CATTLE_CLUSTER_PROVISION_TIMEOUTkubectl get cluster <name> -o yaml | grep provisioningTimeoutagent registration timeout to confirm the active timeout.Active or Failed to validate the effective timeout.Provisioning.Need to confirm the exact Rancher version you are running? The default timeout and env‑var name are stable across 2.6+, but earlier 2.x releases used a different key.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 31 Jan 2026, 06:07 UTC
Rancher’s cluster-agent pod reports a heartbeat to the Manager. If no healthy heartbeat arrives within the agentConnectionTimeout period, the Cluster CRD is marked Failed. The value is stored in the Cluster’s spec.agentConnectionTimeout (seconds). If left unset, the Manager falls back to the global default in cattle-global-configuration.yaml – typically 900 (15 min) in 2.6.x releases.
spec.agentConnectionTimeout in the Cluster resource or add an annotation cluster.cattle.io/agent-connection-timeout.cattle-global-configuration.yaml and change cluster-agent-connection-timeout, then restart the Manager.After changing the value, watch kubectl get cluster -o yaml for the new status and verify with kubectl logs -n cattle-system cattle-cluster-agent that heartbeats resume.