Kubernetes HPA: Scaling on CPU Without the Flapping and the Foot-Guns

Your service handles 200 requests per second on a quiet Tuesday, then a marketing email goes out and traffic triples. If you sized your Deployment for peak, you're paying for idle pods most of the day. If you sized it for average, the spike melts it. The Horizontal Pod Autoscaler (HPA) exists to close that gap: it watches a metric and adjusts the replica count for you. The catch is that HPA only works well if you understand what it's actually measuring and where it can mislead you.

The takeaway: HPA is reliable for CPU-driven scaling when your pods have honest resource requests, but it reacts on a delay, depends on the Metrics Server, and can thrash or misbehave if you skip the stabilization settings or pick the wrong metric.

How the control loop actually decides

HPA is a control loop, not an event handler. Roughly every 15 seconds (configurable via --horizontal-pod-autoscaler-sync-period on the controller manager), it queries the Metrics Server, compares observed usage to your target, and computes a new replica count:

desiredReplicas = ceil[currentReplicas * (currentMetricValue / desiredMetricValue)]

Two things follow from that formula. First, the target is a percentage of the pod's CPU request, not of the node. If your container requests 250m CPU and you set averageUtilization: 50, scaling kicks in when average usage across pods exceeds 125m — regardless of how big the node is. Second, because the formula multiplies the current replica count, a deployment running at 2 replicas scales more coarsely than one at 10. Small replica counts plus aggressive targets produce jumpy behavior.

This is why the two hard prerequisites matter so much:

• Metrics Server must be installed. On managed clusters (EKS, GKE, AKS) it usually is; on kubeadm-built clusters it often isn't. Check with kubectl top pods — if that errors, HPA has nothing to read.
• Every container in the pod needs a CPU request. Utilization is computed against requests, so a pod without one makes the math undefined and the HPA will sit there reporting <unknown>.

A worked example: scaling a web deployment

Here's a complete, minimal setup. Run these from any machine with kubectl access to the cluster; you need permission to create Deployments, Services, and HPAs in the target namespace.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: your-registry/web:1.4.2
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
          limits:
            memory: 512Mi
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

Apply it, then verify the HPA is reading metrics before you trust it:

kubectl apply -f web-hpa.yaml
kubectl get hpa web

The TARGETS column should show something like 12%/60% within a minute or two. If it shows <unknown>/60% for more than a couple of minutes, the usual suspects are a missing Metrics Server or a missing CPU request.

To see it actually scale, generate load from inside the cluster (replace the service name as appropriate):

kubectl run loadgen --rm -it --image=busybox:1.36 -- /bin/sh
# inside the pod:
while true; do wget -q -O- http://web.default.svc.cluster.local/ > /dev/null; done

In a second terminal, watch the replica count move:

kubectl get hpa web -w

You should see utilization climb past 60%, then REPLICAS step up. Kill the load generator and, after a few minutes, watch it scale back down. Note the asymmetry — that delay is deliberate, and it's the next section.

Stopping the flap: stabilization windows

Metrics are noisy. A 10-second CPU spike shouldn't trigger a scale-up that takes 90 seconds to become ready, and a brief lull shouldn't tear down capacity you're about to need. HPA handles this with stabilization windows, configured under behavior in the autoscaling/v2 API:

  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Pods
        value: 4
        periodSeconds: 60

The defaults are already sensible: scale up immediately (window of 0), scale down cautiously (5-minute window, remembering the highest recommendation in that period). The scale-down policy above adds a rate limit — never remove more than 50% of pods per minute — which protects you from a metric glitch halving your fleet. Tune scale-up aggressively and scale-down conservatively; the cost of an extra pod for five minutes is trivial compared to the cost of dropping capacity mid-burst.

Where CPU-based HPA lets you down

It's reactive, not predictive. The loop needs to observe elevated usage, decide, create pods, and wait for them to pass readiness. For sudden 10x spikes, that lag means some requests will fail or queue. Mitigations are lower targets (scale earlier), pre-scaling before known events, or pairing HPA with the Cluster Autoscaler so new pods actually have nodes to land on — otherwise they sit Pending and the HPA looks like it's doing nothing.

Memory is a trap for garbage-collected runtimes. A JVM or Go service under load may grow heap and not return it promptly, so memory utilization stays high after the spike ends. HPA then scales up on stale signal and never scales down. CPU is usually the safer default metric; if your bottleneck is genuinely memory, set a generous target and test the scale-down path explicitly.

CPU isn't always the bottleneck. If your service is I/O-bound or limited by a downstream database, CPU can sit at 20% while latency explodes. That's the case for custom metrics — scaling on queue depth or request rate via the Custom Metrics API (e.g., Prometheus Adapter) — but that's a meaningfully bigger operational commitment than the built-in resource metrics.

Requests that don't match reality distort everything. If your CPU request is 250m but the app comfortably uses 2 cores, a 60% target means you're permanently underscaled relative to actual demand. Load-test to find real usage, then set requests honestly.

Before you call it done

Three checks worth doing in a staging environment: confirm kubectl get hpa shows real percentages rather than <unknown>; run a load test and verify pods scale up and back down within your expected window; and describe the HPA (kubectl describe hpa web) to read its event history, which tells you exactly why each scaling decision was made. If all three behave, you have an autoscaler you can trust — sized by your requests, damped against noise, and honest about its limits.