Scaling AKS Node Pools with the Cluster Autoscaler

Solving Resource Exhaustion with Automatic Scaling

When a Kubernetes pod cannot be scheduled because no node has enough available CPU or memory, it enters a Pending state. Manually increasing the node count in an Azure Kubernetes Service (AKS) node pool is a reactive approach that often leads to either wasted spend (over-provisioning) or application downtime (under-provisioning).

The AKS Cluster Autoscaler solves this by monitoring the Kubernetes scheduler. When it detects pods that are unschedulable due to resource constraints, it automatically requests the underlying Virtual Machine Scale Set (VMSS) to add nodes. Conversely, it removes underutilized nodes after a stabilization period to reduce costs.

Enabling and Configuring the Autoscaler

The Cluster Autoscaler is an add-on that can be enabled during cluster creation or applied to an existing cluster. It requires a defined minimum and maximum node count to establish the scaling boundaries.

Run the following command from a terminal with Azure CLI installed and the appropriate contributor permissions for the resource group:

# Enable autoscaling on an existing AKS cluster
az aks update \
  --resource-group <MyResourceGroup> \
  --name <MyAKSCluster> \
  --enable-cluster-autoscaler \
  --min-count 1 \
  --max-count 5

Expected Result: The command updates the cluster profile. You can verify the status by running:

az aks show --resource-group <MyResourceGroup> --name <MyAKSCluster> --query addonProfiles.clusterAutoscaler.enabled

The output should return true.

Testing the Scale-Out Mechanism

To verify the autoscaler is functioning, deploy a pod that requests more resources than any single remaining node can provide. For example, if your nodes have 2 vCPUs, request 4 vCPUs to force the scheduler to trigger a scale-out event.

# Run a pod with high resource requests to trigger scaling
kubectl run scale-test --image=busybox --restart=Never \ 
  --requests='cpu=4,memory=8Gi' -- /bin/sleep 3600

Run watch kubectl get nodes in a separate terminal. You will see the pod remain Pending for several minutes while the VMSS provisions a new node. Once the node joins the cluster, the pod will transition to Running.

Operational Limits and Constraints

Constraint	Impact	Mitigation
Subnet IP Space	If the VNet subnet is full, the autoscaler cannot add nodes despite having quota.	Ensure the subnet CIDR is large enough to accommodate the max-count.
Metric Blindness	The autoscaler reacts to pod requests, not actual CPU/RAM usage or queue depth.	Use the Horizontal Pod Autoscaler (HPA) to increase pod replicas based on metrics.
Scale-in Delay	Nodes are not removed instantly; there is a stabilization window (typically 10 minutes).	Expect a lag between traffic drop and cost reduction.
Virtual Nodes	Pods scheduled on Azure Container Instances (ACI) do not trigger VM scaling.	Configure separate scaling logic for ACI-based workloads.

Common Configuration Mistakes

• Matching Min/Max Counts: Setting --min-count 3 --max-count 3 effectively disables the autoscaler, as the cluster has no room to grow or shrink.
• Taint/Selector Mismatches: If you apply a taint to a node pool but the pending pods do not have the corresponding toleration, the autoscaler may add nodes that the pods still cannot use, leading to a loop of failed scaling.
• Ignoring Pod Disruption Budgets (PDBs): During scale-in, the autoscaler may attempt to remove a node. If a PDB is too restrictive, the autoscaler cannot evict the pods, and the node will remain active, preventing cost savings.

Diagnostic Verification

If nodes are not scaling as expected, check the cluster events. The Cluster Autoscaler logs its decisions directly to the Kubernetes event stream.

# Filter events for scaling activity
kubectl get events --field-selector reason=ScalingUp,ScalingDown

Look for messages indicating noScaleUp (often due to reaching max-count) or ScaleUp (indicating a request was sent to Azure).

Rollback Procedure

To disable the autoscaler and return to a fixed node count, run the following command. Note that this will stop automatic scaling but will not automatically shrink the cluster to your previous manual size; you must manually adjust the count if necessary.

az aks update --resource-group <MyResourceGroup> --name <MyAKSCluster> --disable-cluster-autoscaler