Scaling AKS Node Pools with Azure Monitor Custom Metrics

Problem: Pods wait for nodes while traffic spikes

Your application runs in Azure Kubernetes Service (AKS) and experiences sudden bursts of traffic. The Horizontal Pod Autoscaler (HPA) can add pods based on CPU or memory, but if the cluster lacks free capacity, new pods stay in a pending state until nodes are added. Relying only on resource‑based metrics means you react after pods are already starving, which can increase latency and waste over‑provisioned nodes.

Thesis: Use Azure Monitor custom metrics to drive proactive node‑pool scaling

By exposing application‑specific signals (e.g., request latency, queue depth) to Azure Monitor and feeding them into the cluster autoscaler through the Azure Monitor metrics adapter, you can trigger node additions before pods become unschedulable. This approach reduces pod startup latency and keeps idle capacity low.

How the pieces fit together

• Application emits a Prometheus‑style metric (e.g., http_requests_per_second) to a sidecar or agent.
• Azure Monitor metrics adapter scrapes that metric and makes it available via the custom.metrics.k8s.io API.
• Horizontal Pod Autoscaler targets the custom metric and scales pods when the metric exceeds a threshold.
• Cluster autoscaler watches pod scheduling; when pods cannot be placed due to insufficient node capacity, it requests more nodes from the node pool.

Worked example: scaling based on request rate

1. Prerequisites
   • AKS cluster (Kubernetes 1.23+)
   • Azure CLI az version 2.50+
   • Permissions: Microsoft.ContainerService/managedClusters/read and Microsoft.Insights/metrics/write on the cluster’s resource group.
2. Deploy the metrics adapter

   # Add the Azure Monitor repo
   helm repo add azure-monitor https://azure.github.io/azure-monitor-charts
   helm repo update
   
   # Install the adapter (replace placeholders)
   helm install azure-monitor-adapter azure-monitor/azuremonitor-containers-metrics-adapter \\
     --namespace kube-system \\
     --set azureMonitor.auth.type=managedIdentity \\
     --set azureMonitor.auth.clientId=<USER_ASSIGNED_IDENTITY_CLIENT_ID> \\
     --set azureMonitor.auth.resourceId=<AKS_CLUSTER_RESOURCE_ID> \\
     --set prometheus.scrape.enabled=true
   

   Run the command in a terminal with Azure CLI logged in (az login). The command creates a Deployment and Service in kube-system. Verify with kubectl get pods -n kube-system -l app=azuremonitor-containers-metrics-adapter.

3. Expose a test custom metric

   For demonstration, deploy a simple sidecar that increments a Prometheus counter every second.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: metric-generator
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: metric-generator
     template:
       metadata:
         labels:
           app: metric-generator
       spec:
         containers:
         - name: generator
           image: mcr.microsoft.com/oss/kubernetes/pause:3.6
           command: [\"/bin/sh\",\"-c\",\"while true; do echo \\\"http_requests_per_second 1\\\" >> /tmp/metrics.prom; sleep 1; done\"]
           volumeMounts:
           - name: metrics
             mountPath: /tmp
         - name: exporter
           image: prom/node-exporter:v1.5.0
           args: [\"--collector.textfile.directory=/tmp\"]
           volumeMounts:
           - name: metrics
             mountPath: /tmp
         volumes:
         - name: metrics
           emptyDir: {}
   

   Apply with kubectl apply -f metric-generator.yaml. The adapter will scrape the http_requests_per_second metric and expose it as custom.metrics.k8s.io/http_requests_per_second.

4. Create an HPA that targets the custom metric

   kubectl autoscale deployment metric-generator \\\
     --cpu-percent=50 \\\
     --min=1 \\\
     --max=10 \\\
     --custom-metrics-name=http_requests_per_second \\\
     --custom-metrics-target=AverageValue=50
   

   This tells the HPA to add pods when the average request rate per pod exceeds 50 req/s.

5. Generate load to observe scaling

   Increase the generator’s output (edit the Deployment to change the echo value to 200) and watch:

   kubectl get hpa -w
   kubectl get nodes -w
   

   You should see the pod count rise, then after a short delay (typically 1‑2 minutes due to metric ingestion latency) the node count increase as the cluster autoscaler adds nodes to accommodate the new pods.

6. Verify in Azure Monitor

   Open Metrics Explorer, select the AKS cluster resource, namespace default, metric http_requests_per_second. Confirm the value spikes and that autoscale events appear in the cluster autoscaler logs (kubectl logs -n kube-system -l app=cluster-autoscaler).

Trade‑off and limitation

The custom metric pipeline introduces latency (usually 1‑2 minutes) because Azure Monitor must ingest, aggregate, and expose the metric before the HPA can act. If your application can experience sub‑minute spikes, set the HPA threshold with a safety buffer or supplement with resource‑based scaling for fast reaction. Additionally, the metrics adapter requires managed‑identity access to Azure Monitor; mis‑configured permissions silently drop metrics, so verify with kubectl logs -n kube-system -l app=azuremonitor-containers-metrics-adapter for 403 errors.

Actionable closing

Start small: deploy the adapter, expose a single application metric, and create an HPA that targets it. Monitor node count and pod latency for a week, then adjust the metric threshold and stabilization window to match your traffic patterns. This gives you workload‑aware scaling that reduces over‑provisioned nodes while keeping pod startup latency low.