Choosing Between HPA and VPA in Kubernetes: A Practical Decision Guide

Problem Statement

In a production Kubernetes cluster you often need to adapt to changing demand. The two built‑in autoscaling mechanisms – Horizontal Pod Autoscaler (HPA) and Vertical Pod Autoscaler (VPA) – both claim to keep workloads healthy, but they do it in fundamentally different ways. Choosing the wrong one can lead to wasted resources, unexpected restarts, or service outages.

Decision Context

When deciding between HPA and VPA you should evaluate:

• Workload type: stateless vs stateful, single‑instance vs multiple‑instance.
• Resource usage patterns: bursty, steady, or highly variable.
• Latency tolerance: can the service handle a pod restart or a temporary spike in replica count?
• Operational complexity: how much extra plumbing (admission controllers, metrics server) is acceptable?
• Cluster capacity constraints: can you afford more nodes or more memory per pod?

Option Comparison

Feature	HPA	VPA
What it changes	Replica count	CPU/Memory requests & limits
Typical workloads	Stateless HTTP services, micro‑services	Stateful databases, CPU‑bound batch jobs
Metric source	External metrics (CPU, memory, custom)	Historical usage collected by VPA admission controller
Impact on pods	Creates new pods, no restarts of existing ones	May restart pods to apply new resource values
Operational overhead	Single YAML, requires metrics-server	Deploy VPA admission controller, monitor suggested values
Conflict risk when combined	None	VPA can override HPA if update mode is not set correctly
Cold start / latency	Potentially higher due to increased pod count	Minimal, but restarts can interrupt service
Cluster capacity impact	Increases node usage proportionally to replica count	Optimizes per‑pod resource usage, can free nodes

Trade‑Off Summary

• HPA is simple to enable and scales out quickly, but it can overshoot during metric spikes, leading to resource contention.
• VPA fine‑tunes pod resources, reducing waste, yet it may trigger pod restarts, which is problematic for high‑availability services.
• Running both together can deliver the best of both worlds, but you must set VPA’s updateMode to Off or Initial to avoid conflicts.

Concrete Implementation & Validation

Prerequisites

• kubectl 1.28+ configured for your cluster.
• Cluster has metrics-server deployed for HPA.
• VPA admission controller is running (e.g., vpa-admission-controller).

Step 1 – Deploy a Sample Stateless Service

# Create a simple Nginx deployment
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: nginx
          image: nginx:stable
          resources:
            requests:
              cpu: 200m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi
EOF

Step 2 – Enable Horizontal Pod Autoscaler

# Scale based on CPU usage
kubectl autoscale deployment demo-app \
  --cpu-percent=50 \
  --min=2 \
  --max=10

Check scaling in real time:

kubectl get pods -l app=demo -w

Generate load:

# In a separate terminal
kubectl run -i --tty load-generator --image=busybox --restart=Never -- sh -c "while true; do wget -q -O- http://demo-app.default.svc.cluster.local; sleep 1; done"

Observe pods being created or terminated as CPU demand rises.

Step 3 – Deploy Vertical Pod Autoscaler

# Create VPA object
kubectl apply -f - <<'EOF'
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: demo-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: demo-app
  updatePolicy:
    updateMode: Off  # Prevent VPA from restarting pods automatically
  resourcePolicy:
    containerPolicies:
      - containerName: nginx
        controlledResources: ["cpu", "memory"]
EOF

VPA will now suggest new resource requests but will not restart pods because updateMode: Off. To apply the suggestions manually:

# Review suggestions
kubectl get vpa demo-vpa -o yaml | grep -A3 "recommendation"

# Update deployment manually
kubectl patch deployment demo-app -p '{"spec":{"template":{"spec":{"containers":[{"name":"nginx","resources":{"requests":{"cpu":"250m","memory":"192Mi"}}}]}}}}'

Step 4 – Verify Combined Behavior

Now enable both autoscalers and monitor:

# Ensure VPA does not interfere
kubectl patch verticalpodautoscaler demo-vpa -p '{"spec":{"updatePolicy":{"updateMode":"Off"}}}'

# Observe pod counts (HPA) and resource requests (VPA)
watch -n1 "kubectl get pods -l app=demo -o wide"
watch -n1 "kubectl describe deployment demo-app | grep -A4 'Resources'"

In the logs of the VPA controller you can see the recommendation flow:

kubectl logs -n kube-system vpa-controller | grep -i 'recommendation'

Validation Checklist

• HPA: Pod count changes in response to CPU spikes.
• VPA: Suggested resource values appear in the VPA status; manual patch updates the deployment.
• Combined: No pod restarts due to VPA, and replica count still scales.
• No resource contention: monitor node usage with kubectl top nodes.

Operational Considerations

• Metrics Server: HPA requires metrics-server; VPA can operate without external metrics but relies on its own historical data.
• Restart impact: For services that cannot tolerate downtime, keep VPA updateMode: Off and apply recommendations manually.
• Cooldown periods: Configure --horizontal-pod-autoscaler-sync-period and VPA’s --recommender-sync-period to avoid rapid oscillations.
• Resource limits: When using VPA, always set hard limits to prevent runaway memory usage.

Conclusion

Use HPA when you need rapid scaling of stateless services and can tolerate increased pod counts. Use VPA for stateful or resource‑tight workloads where you want to reduce per‑pod waste, but be mindful of restarts. If your workload benefits from both, enable both autoscalers, but set VPA’s updateMode to Off or Initial to avoid conflicts. Regularly review VPA recommendations and adjust thresholds to keep your cluster efficient and stable.