Talos worker nodes on spot instances not scaling down during idle periods

Recommended Scale-Down Settings

For a mostly idle workload on spot-backed Talos workers, configure the cluster-autoscaler with these flags:

• --scale-down-utilization-threshold=0.3 (30%) — lower than the default 50% so nodes with minimal pods become candidates.
• --scale-down-unneeded-time=20m — extended from the default 10m to avoid removing nodes during transient dips.
• --scale-down-delay-after-add=10m and --scale-down-delay-after-delete=10m — prevent thrashing after scaling events.
• --scale-down-delay-after-failure=3m — safe default.
• --scale-down-unready-time=5m — lets the autoscaler act on unready nodes (e.g., spot termination notice) without waiting for utilization checks.

These values assume cluster-autoscaler v1.28+ and a single cloud provider. If you run mixed instance types in one ASG, split into separate node groups per instance type; the autoscaler's unneeded-time logic does not account for instance diversity.

Verification Before Committing

1. Deploy a test Deployment with 3 replicas and a PodDisruptionBudget: minAvailable: 2.
2. Simulate scale-down by scaling the autoscaler deployment to zero in its namespace: kubectl scale deployment cluster-autoscaler --replicas=0 -n kube-system.
3. Observe pod rescheduling latency and PDB enforcement: kubectl get events -w --field-selector reason=Evicted.
4. Alternatively, manually cordon and drain a worker: kubectl cordon <node> && kubectl drain <node> --ignore-daemonsets --delete-emptydir-data.
5. Run a 24-hour soak test with a pause-container Deployment (3 replicas, PDB minAvailable=2) and monitor node count: kubectl get nodes -w.

Talos Machine Configuration for Graceful Drain

Talos has no dedicated drain timeout; drain behavior is driven by the kubelet's --pod-eviction-timeout (default 5m) and your PodDisruptionBudgets. Set the kubelet flag via machine config:

machine:
  kubelet:
    extraArgs:
      pod-eviction-timeout: "5m0s"

Verify with talosctl get machineconfig -n <worker-ip> | grep pod-eviction-timeout. Also ensure:

• cluster.allowSchedulingOnControlPlanes: false — control planes must not be spot-backed.
• Worker nodes carry topology.kubernetes.io/zone labels for zone-aware scaling.

Spot-Specific Guardrails

Install the cloud provider's spot termination handler as a DaemonSet (AWS Node Termination Handler, GCP Spot Termination Notifier, Azure Spot Termination Notifier). It cordons and drains nodes on the 2-minute notice, complementing autoscaler decisions. Confirm it runs: kubectl get ds -n kube-system | grep -i termination.

Without this handler, PDBs do not protect against involuntary spot termination.

One Missing Diagnostic Detail

What cloud provider and instance group / ASG configuration are you using? The autoscaler's --nodes spec must match your group's min/max exactly, and some providers (e.g., AWS mixed-instances policies) require separate node groups per instance type for predictable scale-down.