PVC vs object store artifact storage for Kubeflow Pipelines with limited ops overhead

Question

Kubeflow is an ML platform for pipelines, notebooks, Katib and model serving on Kubernetes. Zero-downtime migration primitives for general applications are outside its scope.

The design question is for Kubeflow Pipelines artifact and metadata persistence in a small single-cluster setup with limited operational overhead. The documented approaches are per-namespace PersistentVolumeClaim storage for simplicity and cost, versus external object storage such as S3 or MinIO for durability and portability.

PVC storage reduces external dependencies but ties runs to the cluster and node, with manual backup and restore. Object store decouples artifacts from cluster lifecycle and enables sharing across namespaces and clusters, but adds credential management, network egress and operational overhead.

Artifact storage is typically a cluster-level decision that affects all namespaces, and changing it later requires reconfiguration with no built-in migration tooling. Behavior is version-sensitive across SDK and Pipelines releases.

Which storage mode preserves pipeline run artifacts after pod deletion in a single namespace under current Kubeflow Pipelines configuration? What operational changes are required to switch artifact storage later without built-in migration tooling? How do SDK v1 and v2 component models differ in artifact handling for the two storage options?

Wasi · Answer

Answer
Object storage preserves pipeline run artifacts after pod deletion in a single namespace. A PersistentVolumeClaim also preserves them only if the underlying PersistentVolume uses a Retain reclaim policy and outlives the pod and namespace; ephemeral or delete-on-removal volumes do not. This concerns artifact bytes, while MLMD metadata is a separate database concern.
Likely explanation
If artifacts disappear after a run, the usual cause is the volume reclaim policy or an ephemeral mount, not Kubeflow Pipelines itself. Object storage removes that failure mode by decoupling the bucket from the cluster lifecycle.
Confirmed facts

Artifact persistence is controlled by the artifact-store backend and retention policy, not by pod lifecycle.
Object storage keeps artifacts available after pod deletion, cluster upgrades, or namespace teardown while the bucket and credentials remain valid.
A PVC preserves artifacts only when the PersistentVolume is retained and backed up; node-bound storage classes or Delete reclaim policies can lose data.
Kubeflow Pipelines has no built-in artifact-store migration tool, so changing stores requires reconfiguration and manual copying of historical artifacts.
SDK v1 compiles pipelines to Argo YAML and handles artifacts mainly as file paths and URIs uploaded to the configured store. SDK v2 uses typed Input[Artifact] and Output[Artifact], an intermediate representation, and MLMD tracking; artifact URIs and metadata behavior are release-sensitive.

Steps for this case

Inspect the KFP deployment for the artifact-store bucket or PVC mount and confirm the PersistentVolume reclaim policy; do not print secret values.
Run one test pipeline, list artifacts, delete the producing pod, then read the artifact URI to confirm persistence independent of pod deletion.
For PVC, use read-only checks such as kubectl get pv,pvc -n NAMESPACE -o wide and prefer Retain reclaim policy with scheduled volume backups.
To switch later, provision the new bucket or PVC, update the KFP deployment and credentials for the new endpoint, run new pipelines against the new store, manually copy historical artifacts if old runs must remain readable, and keep the old store reachable until verification is complete. The change is cluster or profile level, not isolated to one namespace.

Verification
Use read-only checks: kubectl get pv,pvc -n NAMESPACE -o wide, kubectl get secret -n NAMESPACE SECRET-NAME -o jsonpath='{.metadata.name}', and pip show kfp | sed -n 's/^Version: //p'. Confirm the SDK version and KFP release before writing component specs, because v1 and v2 artifact declarations are not interchangeable.
Missing diagnostic that changes the exact procedure: the installed KFP SDK version and release. The recommendation does not change, but component syntax and migration validation steps do.

PVC vs object store artifact storage for Kubeflow Pipelines with limited ops overhead

1 answer

Answer

Likely explanation

Confirmed facts

Steps for this case

Verification

Your perspective belongs here.

1 question comment

Operational Considerations