Scaling Microservices with Kubernetes HPA and Custom Metrics

The problem: CPU‑only scaling misses the real load

Most teams start with the Horizontal Pod Autoscaler (HPA) watching CPU or memory. That works for compute‑bound workloads, but a request‑heavy API often saturates latency or queue depth long before CPU hits the threshold. The result is either over‑provisioned pods or painful latency spikes.

Why custom metrics change the game

Kubernetes HPA can consume any metric exposed through the custom.metrics.k8s.io API. By wiring a Prometheus Adapter (or another metrics adapter) you can scale on application‑specific signals such as requests‑per‑second, 95th‑percentile latency, or a business KPI like "orders‑per‑minute". The HPA controller polls the metric every 15 seconds (configurable via --horizontal-pod-autoscaler-sync-period) and adjusts replica count to keep the metric at a target value.

Worked example: Go HTTP service scaled by RPS

1. Deploy the service and expose a Prometheus scrape target

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: go-api
  template:
    metadata:
      labels:
        app: go-api
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      containers:
      - name: server
        image: ghcr.io/example/go-api:v1.2.0
        ports:
        - containerPort: 8080
        env:
        - name: PORT
          value: "8080"
--- 
apiVersion: v1
kind: Service
metadata:
  name: go-api
spec:
  selector:
    app: go-api
  ports:
  - port: 80
    targetPort: 8080

The pod emits a http_requests_total counter on /metrics. Prometheus scrapes it every 15 s.

2. Install Prometheus Adapter for custom metrics

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus-adapter prometheus-community/kube-prometheus-stack \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheusAdapter.enabled=true \
  --set prometheusAdapter.config.rules[0].seriesQuery='http_requests_total{namespace!="",pod!=""}' \
  --set prometheusAdapter.config.rules[0].resources[0].name="http_requests_per_second" \
  --set prometheusAdapter.config.rules[0].resources[0].targetAverageValue="100"

Run this with cluster‑admin rights. The adapter now exposes http_requests_per_second via the custom metrics API.

3. Create the HPA that targets 100 RPS per pod

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: go-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: go-api
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "100"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
    scaleUp:
      stabilizationWindowSeconds: 60

Apply with kubectl apply -f hpa.yaml (requires create/patch on horizontalpodautoscalers in the namespace).

4. Verify scaling works

1. kubectl get hpa go-api-hpa -w – watch the TARGETS column; it should show current RPS vs. 100.
2. Generate load: hey -z 2m -c 50 http://go-api.example.com/ (run from a pod with network access).
3. Observe kubectl get pods -l app=go-api – replica count should rise until each pod averages ~100 RPS.
4. Check events: kubectl describe hpa go-api-hpa for any FailedGetPodsMetric or adapter errors.

Trade‑offs and limitations

• Metric latency: The 15 s polling interval plus Prometheus scrape latency means scaling reacts ~30‑45 s after a spike. For sub‑second SLAs you may need a faster adapter or a custom controller.
• Noisy metrics cause thrashing: Bursty counters can make HPA oscillate. Use behavior.scaleDown.stabilizationWindowSeconds (as shown) and consider recording rules that smooth the rate (e.g., rate(http_requests_total[30s])).
• Stateful workloads: Scaling a database or cache with HPA risks split‑brain or data loss. Prefer StatefulSet with manual scaling or an operator that understands replication.
• Adapter dependency: If the Prometheus Adapter pod crashes, HPA falls back to unknown metric and stops scaling. Run the adapter with a Deployment of at least two replicas and a PodDisruptionBudget.

Actionable next steps

1. Identify the single most representative load metric for each service (RPS, queue depth, latency percentile).
2. Add a Prometheus recording rule that computes a per‑pod average over a 30‑second window.
3. Deploy the Prometheus Adapter (or your preferred custom‑metrics server) with high availability.
4. Create an HPA per service using AverageValue targets; tune minReplicas/maxReplicas and stabilization windows.
5. Automate verification in CI: a test that applies the HPA, fires a load generator, and asserts replica count changes within an expected window.

With these pieces in place, your autoscaling decisions reflect the actual work your code does—not just the CPU it burns.