Question
Duplicate rows after Kubeflow Pipelines component retry
Rin Indigo
0 reputation · 15 Nov 2025, 11:00 UTC
123.3K views1
Loading…
0 reputation · 15 Nov 2025, 11:00 UTC
Goal: configure a Kubeflow Pipelines component retry policy without duplicating writes to external storage such as SQL, S3, or GCS. Assumes Kubeflow Pipelines SDK v2 with the default Argo-based orchestrator.
The KFP orchestrator restarts the container on retry but does not roll back side effects already committed by the previous attempt. Idempotency must therefore exist inside the component logic.
One unresolved behavior is the state of partially completed outputs when a pod is preempted compared with a container that exits non-zero during the same retry cycle. This distinction matters for deciding whether a retry can safely resume or must restart the write from the beginning.
Does the KFP SDK expose retry-level metadata that distinguishes preemption from application failure? Can component authors rely on the output artifact manifest to detect partial writes before retrying? What is the recommended pattern for making writes safe across KFP retries?
34775 reputation · 15 Nov 2025, 14:19 UTC
The KFP SDK does not expose specific retry-level metadata to the container that distinguishes a Kubernetes preemption from an application-level non-zero exit. In both cases, the orchestrator perceives the pod as failed and triggers the retry policy.
Furthermore, component authors cannot rely on the output artifact manifest to detect partial writes. The manifest is only updated by the KFP engine after a container exits successfully with code 0. If a pod is preempted halfway through a write, the manifest will not reflect the partial state, yet the data may already exist in your external system.
Since the environment cannot guarantee a clean state upon retry, the component must handle existing data using the following strategies:
kubeflowUPSERT operations or ON CONFLICT clauses. Utilize a combination of the KFP Run ID and Task ID as a unique constraint to ensure that a retried task updates the existing record rather than creating duplicates.To verify your component is safe, manually trigger a failure mid-write (e.g., using an `exit 1` after partial upload) and observe that the subsequent retry does not result in duplicate entries in your target storage.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 15 Nov 2025, 12:21 UTC
Argo-based KFP retries restart the component container from scratch, so in-process state, open files and local side effects are lost between attempts; the orchestrator does not automatically roll back external writes already committed to SQL, S3, or GCS. Output artifacts are uploaded only after a successful container exit, meaning a retry begins with a fresh working directory and no partial manifest is available for inspection. To prevent duplicate rows, component authors should make external storage operations idempotent using deterministic keys and upsert semantics, or stage writes locally then atomically publish after success, letting KFP artifacts represent the final logical state.