Kubeflow Pipelines v2 Artifact Passing and Object Store Latency

KFP v2 Artifact Orchestration

Kubeflow Pipelines (KFP) v2 utilizes a functional DAG model where components execute as isolated containers. Data exchange is managed by persisting output artifacts to an external object store, such as MinIO, which are then referenced as inputs for downstream components.

Performance Constraints

When handling large datasets between components, the abstraction provided by the Pipeline SDK may introduce overhead. There is a technical uncertainty regarding the optimal balance between passing small parameters directly versus using reference-based artifact passing for larger payloads to avoid bottlenecking the orchestration engine.

Given a deployment using KFP v2, what is the recommended threshold for transitioning from parameter-based passing to artifact-based passing to maintain pipeline stability? How does the ML Metadata (MLMD) store impact the latency of these transfers during high-concurrency executions?