Question
Rancher Server ↔ Cluster Agent: Should stale agent pods be auto‑removed on version skew?
Lax Nea
0 reputation · 22 Aug 2022, 23:09 UTC
36.8K views1
Loading…
0 reputation · 22 Aug 2022, 23:09 UTC
Goal: Determine whether Rancher should automatically remove stale cluster‑agent pods when a version mismatch is detected between the Rancher server and a downstream Kubernetes cluster, given that the memory‑leak fix in v2.5.13/v2.6.0 resolved goroutine growth but no automated cleanup exists for version‑skew scenarios.
Constraints: The current behavior leaves old agent pods running until manual deletion or cluster eviction, which can accumulate resources during upgrade windows; it is unclear what triggers or thresholds would be safe for automatic removal without disrupting workloads or violating pod disruption budgets.
Should Rancher introduce an automated eviction policy for cluster‑agent pods that report a version mismatch?
What version‑delta or time‑based criteria should trigger such cleanup to balance safety and resource reclamation?
How would this automated cleanup coexist with existing pod disruption budgets and cluster autoscaler policies to avoid unintended downtime?
33775 reputation · 23 Aug 2022, 05:54 UTC
Rancher does not automatically remove stale cluster-agent pods when a version mismatch is detected, and enabling that behavior by default would be unsafe. Version skew is an expected transient state during rolling upgrades. Deleting an agent pod solely because its image tag differs from the Rancher server risks severing management connectivity for a cluster that is otherwise healthy.
Confirmed: no built-in cleanup mechanism exists for version-skew scenarios. Old agent pods persist until an operator deletes them or the cluster is evicted from Rancher.
Likely explanation: during a Rancher upgrade, downstream agents update asynchronously. A temporary version delta does not mean the pod is stale or leaking resources. If the memory-leak fix in v2.5.13/v2.6.0 resolved goroutine growth, that addresses a symptom but does not establish that version mismatch alone warrants pod removal.
Before changing policy, inspect the current agent state:
rancherkubectl get pods -n cattle-system -l app=cattle-cluster-agentCheck the pod's image tag and recent restarts:
kubectl describe pod <pod-name> -n cattle-systemIn a staging environment, simulate an older agent image and confirm that Rancher logs a version-mismatch warning without deleting the pod. If you later enable an eviction policy, verify that the feature flag is active and that dry-run output matches expectations before enforcement.
Manual cleanup after a confirmed, completed upgrade is safer than automatic eviction. Reserve automation for opt-in, conservative policies that treat version skew as a signal—not a verdict—and always defer to PDBs and cluster health.
Use comments to ask for clarification. Post a solution as an answer.
33,775 reputation · 23 Aug 2022, 11:02 UTC
When considering automated eviction for version skew, it is important to distinguish between active pods and the underlying ReplicaSets. In standard Rancher deployments, the cluster-agent is managed as a Deployment, which typically maintains a revisionHistoryLimit (often defaulting to 10).
While the upgrade controller triggers a rolling update when the desired agent version changes, the "stale" resource footprint often comes from these retained ReplicaSets rather than active pods. If the goal is resource reclamation, adjusting the revisionHistoryLimit may be more effective than a custom eviction policy, as it removes the templates for old versions without risking the stability of the current rollout.
For those verifying current resource usage, you can check the number of retained versions using:
kubectl get rs -n cattle-cluster-agentWould a lower revision limit satisfy the resource concerns, or is the primary issue the persistence of pods during a stalled rolling update?