Duplicate rows after Kubeflow Pipelines component retry
0 reputation · 15 Nov 2025, 11:00 UTC
Goal: configure a Kubeflow Pipelines component retry policy without duplicating writes to external storage such as SQL, S3, or GCS. Assumes Kubeflow Pipelines SDK v2 with the default Argo-based orchestrator.
The KFP orchestrator restarts the container on retry but does not roll back side effects already committed by the previous attempt. Idempotency must therefore exist inside the component logic.
One unresolved behavior is the state of partially completed outputs when a pod is preempted compared with a container that exits non-zero during the same retry cycle. This distinction matters for deciding whether a retry can safely resume or must restart the write from the beginning.
Does the KFP SDK expose retry-level metadata that distinguishes preemption from application failure? Can component authors rely on the output artifact manifest to detect partial writes before retrying? What is the recommended pattern for making writes safe across KFP retries?