Talos worker nodes on spot instances not scaling down during idle periods
0 reputation · 31 Jan 2022, 19:02 UTC
Goal
I am planning a low-traffic Kubernetes workload on Talos Linux and want to keep costs down by running worker nodes on spot/preemptible instances with the Kubernetes cluster-autoscaler handling capacity. The intent is that during idle periods the cluster shrinks to a minimal node count, and scales back out only when real load appears.
Constraints and uncertainty
Talos is immutable and API-driven, so all node lifecycle behavior is declared through machine configuration rather than tuned interactively on the host. There is no SSH or shell access; everything goes through talosctl and the Talos API. Assume a current Talos release and a single-cloud deployment, with the autoscaler deployed as an in-cluster component.
The open decision is how aggressively scale-down should be configured for a workload that is idle most of the time. Spot reclamation is already an availability risk, and an overly eager scale-down policy could compound it by evicting pods onto nodes that may themselves be reclaimed. Conversely, conservative settings may leave nodes running and erase the cost savings that motivated the design. I have not yet confirmed which autoscaler scale-down settings are safe to combine with spot-backed Talos workers, or how to validate the behavior before relying on it.
Questions
- Which cluster-autoscaler scale-down settings (e.g., utilization threshold, delay, cooldown) are appropriate for a mostly idle workload on spot-backed Talos workers?
- What is the recommended way to verify scale-down reschedules pods without downtime before committing to this setup?
- Are there Talos machine configuration options that affect how gracefully a worker drains when the autoscaler or the cloud provider removes it?