Kubeflow Pipelines ↔ KServe: Zero‑Downtime Model Migration During Cluster Upgrade

Goal

To migrate a small Kubeflow application from one cluster to another while keeping model inference traffic continuous, the migration must avoid any interruption during both the Pipelines control‑plane upgrade and the KServe model revision rollout.

Constraints and Uncertainty

Argo Workflows, the engine behind Kubeflow Pipelines, does not expose a documented rolling upgrade path for its API server, which may temporarily block new pipeline submissions. KServe’s traffic‑split mechanism only routes requests to a new revision after that revision passes its readiness probe, but there is no built‑in pre‑warm flag to keep a new container idle until it is fully warmed. The integration boundary between Pipelines and KServe is the artifact ingestion step, currently handled by custom scripts or CI/CD triggers, which may not be fully automated for all storage backends.

Specific Questions

• What mechanisms can be employed in Argo to allow new pipeline submissions to continue while the control plane is being upgraded?
• Does KServe provide a way to pre‑warm a new model revision—such as a flag or readiness hook—to eliminate the cold‑start window during a traffic split?
• Can a CI/CD pipeline be configured to automatically trigger artifact ingestion into KServe after a successful model build, ensuring the traffic split update occurs without manual intervention?