Kubeflow Pipelines and KServe: Zero-Downtime Model Migration Boundary

Answer to the Core Questions

1. Automatic canary rollout on PipelineRun completion? KFP does not ship a built‑in hook that triggers a KServe canary when a PipelineRun finishes. A custom Argo Events rule or a webhook that listens for the Succeeded status is required to call the KServe API and create a canary InferenceService.

2. Lineage during traffic shifts? KServe’s traffic‑splitting (via Istio) only changes the traffic spec; annotations or a model‑registry integration must be used to keep track of which model artifact each split points to.

3. Metadata when moving pipelines across namespaces? Metadata is namespace‑scoped. Moving a PipelineRun or an InferenceService to another namespace does not copy annotations, labels, or custom metadata unless you explicitly migrate those fields or use a shared metadata store.

Practical Steps for a Zero‑Downtime Migration

1. Set up Argo Events
   • Create an EventSource that watches the PipelineRun resource.
   • Define an EventSensor that triggers when the run’s status becomes Succeeded and sends a POST to a KServe webhook endpoint.
2. Implement the KServe webhook
   • Expose a small service (e.g., Flask or FastAPI) inside the cluster that receives the event payload.
   • Use the InferenceService API to create a new canary spec pointing to the freshly built model bundle.
   • Apply a traffic split that starts with 0% for the canary and gradually ramps up to 100% as you monitor metrics.
3. Track lineage explicitly
   • Annotate the InferenceService with a modelRegistryId or similar key that references your model‑registry entry.
   • When traffic is shifted, keep the annotations on both the stable and canary services so that any downstream analytics can correlate traffic with the exact artifact.
4. Namespace migration
   • Use kubectl get pipelineRun -n <old> -o yaml | sed 's/namespace: <old>/namespace: <new>/g' | kubectl apply -n <new> to copy the object.
   • After migration, re‑apply any custom annotations or labels that were lost, or configure a shared ConfigMap/Secret that lives in a common namespace and is referenced by both.

What You Need to Verify

• Confirm your KServe version; some older releases (<0.9) handle canary traffic differently.
• Check that Istio’s VirtualService and DestinationRule reflect the split you expect.
• Validate that the annotations you use for lineage are present on both the stable and canary InferenceServices after a shift.

Missing Diagnostic Detail

Could you share which KServe release you are running? The API shape and canary support differ between 0.8.x and 0.9+, and that may change the exact webhook payload you need to send.