Architecture Note: Using Custom Worker Images in IBM Cloud Kubernetes Service

Requirements

To create and use a custom worker node image in IBM Cloud Kubernetes Service (IKS) you need:

• A base IBM Cloud provided worker image (Ubuntu or CoreOS) that you can pull from IBM Cloud Container Registry (ICR).
• Permission to read images from a private ICR namespace and to update a worker pool (IAM policies: Container Registry - Reader and Kubernetes Service - Operator on the target cluster).
• The ability to build a Docker image that adds the desired packages, kernel modules, or security agents and push it to the same private registry.

Smallest Suitable Design

The minimal workflow consists of three steps: define a Dockerfile that extends the official IBM Cloud worker image, push the resulting image to a private ICR namespace, and instruct the worker pool to use that image.

Dockerfile example

# Use the latest IBM Cloud Ubuntu worker image as the base
FROM icr.io/eks/ibm-cloud-worker:ubuntu-20.04

# Install a monitoring agent (replace with your own software)
RUN apt-get update && apt-get install -y \
    monitoring-agent \
    && rm -rf /var/lib/apt/lists/*

# Ensure the agent starts on boot
COPY monitoring-agent.service /etc/systemd/system/
RUN systemctl enable monitoring-agent.service

Build and push the image (run on a workstation or CI system with Docker and the IBM Cloud CLI installed):

# Log in to IBM Cloud and set the target region/account
ibmcloud login
ibmcloud cr login

# Build the image, substituting your registry, namespace, image name and tag
docker build -t us.icr.io//: .

# Push to the private namespace
docker push us.icr.io//:

Updating the worker pool

With the image available, update the worker pool to reference it. This triggers a rolling update of the nodes in the pool.

ibmcloud ks worker-pool update \
    --cluster  \
    --worker-pool  \
    --image us.icr.io//:

Required permission: the IAM API key used must have ks worker-pool update rights on the cluster.

Trust and Data Boundaries

The custom image runs with the same privileged context as the default IBM Cloud worker image. Consequently, any software baked into the image gains:

• Node‑level access to the container runtime (docker/containerd).
• Ability to read and write the host filesystem.
• Access to the Kubernetes API via the kubelet’s credentials (typically the node’s service account).

Therefore, only trusted code should be included. Image provenance must be verified before deployment, and the image should be scanned for vulnerabilities.

Operational Checks

After the worker pool update completes, verify the rollout with the following checks:

1. Confirm that all nodes report the new image:

   ibmcloud ks workers --cluster 
   

   Look for the IMAGE column; it should show us.icr.io//:.

2. Check that the custom agent is running on a sample node (you can debug into a node or use a DaemonSet):

   # Example using a debug pod to exec into a node
   kubectl debug node/ -it --image=ubuntu
   # Inside the debug container:
   systemctl status monitoring-agent
   

   Expected output: the service should be active (running).

3. Ensure node boot time remains within the provider SLA (typically under 5 minutes). You can approximate this by noting the STATUS transition from provisioning to normal in the worker list.

Failure Modes and Design Triggers

Several conditions can cause the custom image approach to fail or necessitate a redesign:

• Missing hardware drivers: If the image lacks required kernel modules (e.g., for GPUs or specialized NICs), nodes may fail to join the cluster, appearing as critical in ibmcloud ks workers. The remedy is to add the missing modules to the Dockerfile or revert to a base image that includes them.
• Image size limit: IBM Cloud currently rejects worker images larger than 12 GB. Exceeding this limit results in an error during worker-pool update. Keep the image lean by using multi‑stage builds or removing unnecessary packages.
• Insufficient IAM permissions: If the API key lacks read access to the private registry, the update fails with an unauthorized error. Verify the IAM policy includes Container Registry - Reader for the target namespace.
• Base image deprecation: IBM Cloud periodically retires older base images. When a base image reaches end‑of‑life, the worker pool update will be blocked until you rebase your custom image onto a supported release. Monitor the IKS release notes and rebase regularly to avoid forced updates.
• Rolling update capacity impact: Updating the worker image triggers a rolling node replacement. During the update, the pool’s available capacity may dip below the desired level, especially if surge disabled. Schedule changes during low‑traffic windows or enable surge capacity via ibmcloud ks worker-pool update --resize --worker-count .

Practical Verification Steps

To confirm that the custom image is functioning as intended:

1. Run a vulnerability scan before deployment:

   ibmcloud cr va us.icr.io//:
   

2. After rollout, check node readiness and agent status as described in the Operational Checks section.
3. Optionally, deploy a simple DaemonSet that logs the agent’s version to ensure the custom software is present on every node:

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: agent-verification
   spec:
     selector:
       matchLabels:
         app: agent-verification
     template:
       metadata:
         labels:
           app: agent-verification
       spec:
         containers:
         - name: verify
           image: ubuntu
           command: ["sh", "-c", "while true; do systemctl is-active monitoring-agent >> /var/log/agent.log; sleep 60; done"]
           volumeMounts:
           - name: log
             mountPath: /var/log
         volumes:
         - name: log
           hostPath:
             path: /var/log/agent.log
   

These steps give you confidence that the image was deployed correctly, that the custom software is running, and that no immediate operational regressions have been introduced.