Deploying and Verifying HPA v2 in Kubernetes: A Step‑by‑Step Guide

Problem & Takeaway

You have a stateless web service running in a Kubernetes cluster and want to automatically adjust the number of replicas based on CPU usage or a custom metric. The goal is to keep response times within SLA while avoiding over‑provisioning. HPA v2 gives you fine‑grained control, but it requires the right cluster version, metrics, and a correctly written YAML file.

Desired Outcome

1. Cluster runs Kubernetes v1.18+.
2. Metrics‑server (or a custom metrics provider) is deployed and exposing metrics.
3. An HPA v2 resource is created that references the target Deployment.
4. Under sustained load, the Deployment scales up/down as defined.
5. All changes can be rolled back by deleting or editing the HPA.

Prerequisites

• kubectl configured against the target cluster (admin or cluster‑admin role).
• Cluster version ≥ 1.18 (check with kubectl version --short).
• Metrics‑server installed and functional. Verify with kubectl top nodes and kubectl top pods.
• A Deployment named webapp (or replace with your own) that has liveness/readiness probes.
• Optional: Custom metrics source (e.g., Prometheus Adapter) if you want to scale on a metric other than CPU/memory.

YAML Definition – HPA v2

The following example scales the webapp Deployment between 2 and 10 replicas, targeting 80 % CPU utilization. Replace placeholders as needed.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: webapp-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: webapp
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80

For custom metrics, swap the metrics section with something like:

    - type: External
      external:
        metric:
          name: http_requests_per_second
        target:
          type: Value
          averageValue: "1000"

Applying the HPA

1. Save the YAML to hpa.yaml.
2. Run kubectl apply -f hpa.yaml.
   • Requires cluster-admin or autoscaling.k8s.io/hpa privileges.
   • Check output for “created” or “configured”.
3. Verify creation: kubectl get hpa webapp-hpa -o wide.

Verification Under Load

To confirm the HPA reacts to load, generate traffic that keeps CPU above 80 % for a few minutes. Tools like wrk or hey can be used. While the load runs, observe the pod count:

kubectl get pods -l app=webapp -w

Expected behavior: the number of ready pods increases toward maxReplicas as the load persists, then decreases back toward minReplicas when load subsides.

Additional checks:

• kubectl describe hpa webapp-hpa – look for Current Replicas and Desired Replicas.
• kubectl top pod -l app=webapp – confirm CPU metrics are reported and near the target.
• Inspect HPA controller logs if scaling does not occur: kubectl logs -n kube-system -l app=kube-controller-manager.

Common Pitfalls & How to Fix Them

• No metrics available: The HPA will remain at minReplicas. Verify metrics-server is running and kubectl top pods works.
• Custom metric not exposed: Ensure the metrics API is enabled and the adapter exposes the metric name exactly as referenced.
• Pods never become ready: HPA only considers ready pods. Check readiness probes and logs for failures.
• Scaling too aggressively: Setting minReplicas or maxReplicas too high/low can cause resource contention or wasted cost. Adjust conservatively.

Rollback & Recovery

If the HPA causes unintended behavior, you can:

1. Delete the HPA: kubectl delete hpa webapp-hpa. This restores the Deployment to its original replica count.
2. Or edit the HPA to tighten limits: kubectl edit hpa webapp-hpa and adjust minReplicas or maxReplicas.
3. To revert to HPA v1, replace apiVersion: autoscaling/v2 with autoscaling/v1 and remove the metrics block.

Limitations & When to Use v1 Instead

• HPA v2 requires spec.metrics; if you only need CPU or memory, v1 is simpler and avoids custom metrics plumbing.
• Metrics‑server must be present; otherwise the HPA will report “no metrics available” and not scale.
• HPA polls every 15 s by default. Rapid traffic spikes may not be handled instantly.

Practical Checklist

Step	Check
Cluster version	kubectl version --short | grep Server
Metrics‑server	kubectl top nodes
Deployment probes	kubectl describe deployment webapp | grep -i probe
HPA creation	kubectl get hpa webapp-hpa
Scaling under load	kubectl get pods -l app=webapp -w

Conclusion

By following this guide you can confidently deploy an HPA v2 that scales your Deployment based on real‑time metrics, verify its behavior, and roll it back if necessary. Remember to keep your metrics source healthy and set realistic replica bounds to avoid cost spikes or resource contention.