Launch and Monitor a Katib Hyper‑Parameter Tuning Experiment on Kubeflow

Desired Outcome

Deploy a Katib experiment that automatically explores a defined hyper‑parameter space, runs parallel training jobs, and surfaces the best model artifact for downstream use.

Prerequisites

• Kubeflow cluster running 1.8+. Katib API v1beta1 must be available. Verify with kubectl get apiservices and look for v1beta1.katib.kubeflow.org.
• Compute resources. Ensure CPU, memory, and GPU quotas allow the desired parallelism. Use kubectl describe quota to inspect limits.
• Training container image. Image should be accessible from the cluster registry or a public registry. Prefer pre‑pushing large images to avoid startup delays.
• Persistent storage. A PVC or object store (e.g., S3, GCS) mounted to the trial pods for model export.

Step 1: Prepare the Experiment Definition

Katib experiments are defined in a YAML file. The spec references a trial template that contains the training container and command. Hyper‑parameters are declared via a JSON schema.

apiVersion: kubeflow.org/v1beta1
kind: Experiment
metadata:
  name: image-classifier-hpo
spec:
  objective:
    type: minimize
    goal: 0.05
    objectiveMetricName: validation_loss
  parallelTrialCount: 5
  completionCriteria:
    minTrialCount: 10
    maxTrialCount: 20
  algorithm:
    algorithmName: random
  trialTemplate:
    goTemplate:
      metadata:
        name: "trial-{{.TrialIndex}}"
      spec:
        template:
          spec:
            containers:
            - name: trainer
              image: my-registry/trainer:latest
              command: ["python", "train.py"]
              args:
              - "--learning-rate={{.Parameters.learning_rate}}"
              - "--batch-size={{.Parameters.batch_size}}"
              volumeMounts:
              - name: model
                mountPath: /model
        volumes:
        - name: model
          persistentVolumeClaim:
            claimName: model-pvc
  parameterSpec:
    - name: learning_rate
      parameterType: double
      feasibleSpace:
        min: 0.0001
        max: 0.01
        scaleType: log
    - name: batch_size
      parameterType: int
      feasibleSpace:
        min: 32
        max: 256
        scaleType: linear

Replace placeholder values (image, PVC, metric name) with your environment specifics.

Step 2: Submit the Experiment

1. Save the YAML as experiment.yaml.
2. Run:

   kubectl apply -f experiment.yaml
   

3. Confirm creation:

   kubectl get experiments image-classifier-hpo -o wide
   

Step 3: Monitor Execution

• Track experiment status:

  kubectl get experiments image-classifier-hpo -w
  

• List running trials:

  kubectl get trials -o wide
  

• Stream logs for a specific trial job:

  kubectl logs -f job/trial-0
  

  Verify that the training script reports the objective metric (e.g., validation_loss) to stdout or the Katib client library.

Step 4: Retrieve the Best Trial

Katib marks the best trial with Best in the Status column.

kubectl get trials -o wide | grep Best

Alternatively, query the Katib REST API:

curl -s http://katib-api:8080/api/v1beta1/experiments/image-classifier-hpo | jq '.status.trials[] | select(.status == "Best")'

Step 5: Export the Model Artifact

The training container should write the trained model to the mounted volume (/model). After the best trial completes, copy the file to a permanent store:

kubectl cp namespace/trial-0:/model/model.pt /tmp/model.pt

Or, if using an object store, configure the container to upload directly during training.

Step 6: Common Pitfalls & Recovery

• Image pull delays. Pre‑push large images:

  docker pull my-registry/trainer:latest
  docker tag my-registry/trainer:latest my-cluster-registry/trainer:latest
  docker push my-cluster-registry/trainer:latest
  

• Resource limits. If trials stall, reduce parallelTrialCount or increase quota. Use kubectl describe quota to adjust.
• Objective direction. Katib minimizes by default. To maximize, set objective.type: maximize in the spec.
• Trial failures. Inspect logs. If persistent, delete the failing trial:

  kubectl delete trial trial-3
  

• Katib API version mismatch. Verify the API version in the experiment YAML matches the installed Katib API. Older releases use v1alpha1.

Recovery Workflow

1. Abort an experiment:

   kubectl delete experiment image-classifier-hpo
   

2. Confirm all jobs are terminated:

   kubectl get jobs | grep trial | wc -l
   

3. Inspect resource reclamation via kubectl describe pvc model-pvc.
4. Re‑submit with adjusted parameters or resource limits.

Conclusion

By following this guide, you can launch a Katib hyper‑parameter tuning experiment, monitor its progress in real time, and reliably retrieve the best model artifact. Adjust the experiment spec to match your workload, and use the recovery steps to handle common failures without manual intervention.