Kubeflow Pipelines v2 Artifact Passing and Object Store Latency

Direct Answer

For Kubeflow Pipelines v2, a practical rule of thumb is:

• Keep inline parameters under 5–10 KB. Anything larger should be written to an external object store and passed as an artifact reference.
• For data larger than 1 MB, treat it as an artifact; 1–5 MB is a safe upper bound for inline use in most production clusters.

The MLMD store’s performance matters when many concurrent pipelines write metadata records. A local SQLite store can become a bottleneck under high concurrency, adding noticeable latency. Switching to a remote, scalable store such as PostgreSQL, MySQL, or a cloud‑managed database reduces contention and keeps metadata write latency to a few milliseconds per record.

Why the Threshold Matters

KFP v2 serialises component parameters into the pipeline spec. Large values inflate the spec, increase the time the orchestrator spends parsing and validating the DAG, and can trigger time‑outs on the control plane. Artifacts, on the other hand, are streamed directly to the object store; the orchestrator only stores a small URI reference, keeping the control plane lightweight.

Impact of the MLMD Store

During artifact upload, each component writes a metadata record to MLMD. With a local SQLite store, every write is a synchronous disk operation. When dozens of pipelines run in parallel, the SQLite file can become a hot spot, causing metadata writes to stall and downstream components to wait. A remote store distributes the load across a database cluster, so writes are served in parallel and latency stays low even under heavy concurrency.

Practical Verification Steps

1. Deploy a small test pipeline that writes artifacts of sizes 1 KB, 10 KB, 1 MB, and 5 MB.
2. Enable KFP metrics or Prometheus exporters to capture the time from component start to downstream component start.
3. Run >50 concurrent executions of the test pipeline and monitor for timeouts or resource exhaustion.
4. Enable verbose logging for the MLMD store, count the time spent per write operation, and repeat the test with both a local SQLite store and a remote PostgreSQL store.
5. Use kubectl logs or the KFP UI to confirm that large data are stored in the object store and not passed as inline parameters.

Missing Diagnostic Detail

To refine the threshold for your specific cluster, could you share the type and configuration of the object store (e.g., MinIO, GCS, S3) and the current MLMD backend (SQLite vs. PostgreSQL)? This will help tailor the recommendation to your environment.