Stale resource state in Rancher UI during Cluster Agent latency

Rancher utilizes a local cache of downstream cluster resources to minimize direct API pressure on managed Kubernetes clusters. This synchronization depends on the cattle-cluster-agent pushing state updates to the Rancher management plane. In environments with high network latency or intermittent agent connectivity, the Rancher UI may continue to display resources that have already been deleted or modified on the downstream cluster. This creates a discrepancy between the management plane's cached state and the actual cluster state.

Identifying Stale Cached Resources

To identify which resources are stale without querying the downstream API directly, you must inspect the metadata associated with the cluster object itself. Rancher does not provide a per-resource 'stale' flag; instead, it tracks the health of the cluster connection.

• Cluster Heartbeat: Check the lastSeen timestamp of the cluster object in the Rancher API or UI. If the last heartbeat exceeds the expected synchronization interval, all cached resources for that cluster are potentially stale.
• ResourceVersion Mismatch: If the agent is connected but the resourceVersion of a resource in the management plane has not updated despite changes in the downstream cluster, the reconciliation loop is likely bottlenecked.
• Agent Log Analysis: Inspect the cattle-cluster-agent logs on the downstream cluster for 409 errors or connection timeouts, which indicate the agent is unable to push updates to the management plane.

Configurable Thresholds for Staleness

Rancher management plane does not provide a configurable threshold to mark *individual resources* as stale based on the last heartbeat. However, you can configure the cluster-wide heartbeat behavior via the cluster agent settings.

To adjust how quickly Rancher marks a cluster as disconnected, you can modify the cattle-cluster-agent configuration using the following parameters:

• heartbeatInterval: Determines how often the agent sends a heartbeat to the management plane.
• heartbeatTimeout: Determines the duration the management plane waits before marking the cluster as disconnected (which may affect how the UI displays state).

Assumption: This analysis assumes you are using Rancher 2.x or later. In older versions, the reconciliation logic was less transparent and lacked granular heartbeat-based UI indicators.

Diagnostic Request: Are you observing stale state across all resources in the cluster, or only specific types (e.g., Deployments vs Nodes)? This helps determine if the issue is a global agent latency or a specific controller bottleneck.