Diagnosing Pending Pods Caused by IBM Cloud Kubernetes Service Node Pool Autoscaling Failures

Recognizable Condition

Pods remain in the Pending phase and the output of kubectl describe pod shows events such as 0/3 nodes are available: 3 Insufficient cpu or similar memory messages.

Cause and Diagnostic Table

Possible Cause	What to Look For
Autoscaler disabled	No scaling events in autoscaler logs; cluster addon shows disabled
Node pool at maximum size	Current node count equals the defined max size
Pod resource requests exceed node capacity	Sum of requests > allocatable CPU/memory per node
Underlying node failures	Infrastructure events indicate node crashes or unhealthy status

Ordered Checks

1. Verify the cluster autoscaler addon state

   Run the following command in a terminal where the IBM Cloud CLI is installed and you are logged in with an IAM policy that includes Administrator or Operator role on the target cluster:

   ibmcloud ks cluster get --cluster <cluster-name-or-id>

   Look for the field Autoscaler in the output; it should read enabled. If it shows disabled or is missing, the autoscaler is not active.

   Risk: None; this is a read‑only operation.

2. Check autoscaler logs for recent scaling activity

   Retrieve the autoscaler log stream:

   ibmcloud ks cluster logging get --cluster <cluster-name-or-id> --type autoscaler --lines 100

   Search for lines containing scale up or scale down. Absence of such entries over the last 10‑15 minutes suggests the autoscaler is not attempting to add nodes.

   Permission: Same as above; logging read access is required.

3. Compare node pool size to its maximum

   In the IBM Cloud console navigate to Kubernetes > Clusters > <cluster-name> > Node pools or use the CLI:

   ibmcloud ks node-pool get --cluster <cluster-name-or-id> --nodepool <node-pool-name>

   Note the values for Current size and Max size. If they are equal, the pool cannot grow further.

   Risk: Increasing max size may incur additional hourly charges and could be limited by account quotas.

4. Inspect pod resource requests versus node capacity

   Get a representative pod’s YAML (or use kubectl get pod <pod-name> -o yaml) and locate the resources.requests section. Then check the node’s allocatable resources:

   kubectl describe node <node-name> | grep -A 5 'Allocatable'

   If the request for CPU or memory exceeds the node’s allocatable amount, the scheduler cannot place the pod.

   Risk: None; this is observational.

5. Look for infrastructure events indicating node failures

   In the IBM Cloud console, open Monitoring > Activity Tracker or Infrastructure > Events for the cluster’s VPC or classic infrastructure. Filter by the cluster ID and look for entries such as Node power off, Hardware failure, or Instance terminated.

   Risk: None; read‑only.

Fixes Tied to Findings

• Autoscaler disabled – Enable it via the CLI:

  ibmcloud ks feature enable --addon autoscaler --cluster <cluster-name-or-id>

  After enabling, wait a few minutes and re‑check the autoscaler logs for scaling events.

• Node pool at max size – Either raise the max size or add a new node pool:
  • Increase max size (if quota permits):

    ibmcloud ks node-pool update --cluster <cluster-name-or-id> --nodepool <node-pool-name> --size <new-max>

  • Create an additional node pool with a different machine type or zone to spread load:

    ibmcloud ks node-pool create --cluster <cluster-name-or-id> --name <new-pool> --flavor <machine-type> --size <desired-count> --zone <zone>

  Verification: After the change, re‑run the node‑pool get command to confirm the new max size or new pool appears, then watch the autoscaler logs for scale‑up events.

• Pod requests too high – Adjust the pod’s resource specifications:
  • Lower the requests values to fit within node capacity, or
  • If the workload truly needs more resources, consider using a node flavor with larger CPU/memory.

  Update the deployment (or pod) YAML and apply:

  kubectl apply -f <updated-deployment.yaml>

  Risk: Setting requests too low may cause node over‑commit and performance degradation; monitor CPU/memory usage after the change.

• Node failures detected – Recycle the unhealthy nodes or open a support ticket:
  • If the node is still present but unhealthy, you can drain and delete it:

    kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
    kubectl delete node <node-name>

  • The IBM Cloud platform will automatically replace the node if the node pool is managed; otherwise, create a replacement node via the node‑pool update command.

  Verification: After node replacement, check that the node shows Ready status with kubectl get nodes and that pending pods begin to schedule.

Escalation Criteria

Escalate to IBM Cloud Support when any of the following conditions persist after you have performed the checks and fixes above:

• The autoscaler addon shows enabled but the logs contain no scaling attempts for more than 15 minutes.
• You cannot increase the node pool’s max size because the account has reached its quota limit for the selected flavor or region.
• Pending pods remain unscheduled even after correcting resource requests, ensuring the autoscaler is enabled, and confirming that node pools have available capacity.

When opening a ticket, include the cluster ID, node‑pool IDs, relevant autoscaler log snippets, and the output of kubectl describe pod for a representative pending pod.