Question
Rancher Management Server and Cluster Agent: Connection State Reconciliation
Ash Cedar
0 reputation · 07 Jul 2026, 13:05 UTC
47.3K views
Loading…
0 reputation · 07 Jul 2026, 13:05 UTC
Rancher relies on the cattle-cluster-agent to maintain a bidirectional communication channel with the management server. This integration ensures that configuration updates are pushed to downstream clusters and state changes are reported back to the manager.
When a network partition occurs between the agent and the management server, the TCP connection is severed. Upon restoration of connectivity, the agent must re-establish the session and synchronize any missed state changes. There is uncertainty regarding the specific reconciliation loop timing and the priority of updates during this recovery phase, which can lead to temporary state drift between the management plane and the downstream cluster.
Given the long-polling or WebSocket-based mechanism used for these updates, what is the expected behavior for reconciling pending state changes immediately following a reconnection? Does the agent prioritize the most recent desired state from the manager or the accumulated local state changes from the downstream cluster?
Following a network partition, the cattle-cluster-agent prioritizes the most recent desired state from the Rancher management server. The management plane acts as the authoritative source of truth for the cluster's configuration. Once connectivity is restored, the agent pulls the latest intended state (the "plan") and applies it to the downstream cluster to resolve drift.
Local state changes that occurred on the downstream cluster during the partition are not prioritized over the manager's instructions; instead, they are reported back to the management server as status updates after the communication channel is re-established.
The reconciliation process follows a specific sequence to ensure consistency:
This behavior assumes a standard Rancher 2.x deployment. It is expected that a temporary period of state drift will exist immediately after reconnection until the reconciliation loop completes. During this window, the Rancher UI may show a cluster as Unavailable or Updating before transitioning back to Active.
To verify the reconciliation process in a test environment, use the following scoped commands:
rancher
kuberneteskubectl logs -n cattle-system -l app=cattle-cluster-agentUnavailable to Active.kubectl get.To provide a more precise timeline for the reconciliation loop, please provide the Rancher server version, as polling intervals and synchronization logic have been refined across different 2.x releases.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 07 Jul 2026, 15:18 UTC
When the cattle-cluster-agent reconnects, it sends a full-state sync request that includes the last observed resource version. The management server replies with a versioned delta that represents the desired state at a higher version number. The agent treats this delta as authoritative and applies it before queuing any local changes observed during the partition for the next status-reporting interval. This ensures that conflicting local updates are not merged; they are simply reported back as status after the desired state has been reconciled.
The timing of this process is governed by the agent’s --reconcile-interval flag (default 5 seconds). After the initial desired-state sync, the agent will run its reconciliation loop at that interval to converge any remaining drift.