Adopting Kubeflow Pipelines v2: Portable YAML, Local Testing, and Trade‑offs

In the early days of Kubeflow Pipelines, building a workflow meant writing Python code, submitting it to a cluster, and waiting minutes to see if a type mismatch or missing dependency would break the run. This submit‑and‑pray cycle slowed down experimentation for ML engineers.

From Imperative Python to Portable IR

Kubeflow Pipelines SDK v2 (2.x+) changes that model. Instead of the Python runtime interpreting the pipeline, your code acts as a compiler that generates an intermediate representation (IR) — portable Argo Workflow YAML — at authoring time. The same YAML can be sent to any KFP‑compatible backend (the Kubeflow Pipelines service, Vertex AI Pipelines, or a standalone Argo cluster) without rewriting a line of Python.

Components as Typed Building Blocks

In v2 a component is defined with the @dsl.component decorator and explicit type annotations for every input and output. The compiler checks type compatibility before you ever submit the pipeline, catching errors early.

from kfp import dsl
from typing import List

@dsl.component(
    name="preprocess-data",
    image="python:3.9"
)
def preprocess_data(input_path: str) -> List[str]:
    # placeholder logic
    return ["data1.csv", "data2.csv"]

@dsl.pipeline(name="portable-pipeline")
def my_pipeline(data_path: str):
    preprocess_task = preprocess_data(input_path=data_path)

if __name__ == "__main__":
    from kfp import compiler
    compiler.Compiler().compile(pipeline_func=my_pipeline, output_path="pipeline.yaml")

Because the return type is declared as List[str], a downstream component that expects, say, an integer will trigger a compile‑time error, preventing a failed run later.

Local Iteration with the DSL Local Runner

The kfp.dsl.local context lets you execute the same component logic in a plain Python process (or inside a Docker container) without a Kubernetes cluster. This is ideal for quick debugging of algorithms and data‑flow types.

from kfp import dsl

with dsl.local():
    my_pipeline(data_path="/tmp/data")

Note: The local runner does not emulate Kubernetes features such as volume mounting, secret injection, or pod‑level retries. Use it for logic validation, not for final infrastructure testing.

Trade‑off: Static Graphs and Control Flow

Since the pipeline is compiled to a static Argo DAG, ordinary Python for loops over runtime‑determined lists are not allowed. To iterate over a collection produced by an upstream step you must use the SDK primitives dsl.ParallelFor or dsl.If, which compile to specific Argo constructs. Truly dynamic graphs (e.g., recursive pipelines) cannot be expressed; teams needing that behavior must fall back to native Argo Workflows or external orchestration.

Getting Started

1. Install the SDK in an isolated environment: pip install \"kfp>=2.0,<3.0\"
2. Write your components and pipeline using the pattern shown above.
3. Compile to portable YAML: python -c \"from kfp import compiler; compiler.Compiler().compile(pipeline_func=my_pipeline, output_path='pipeline.yaml')\"
4. Validate locally with the dsl.local() runner.
5. Submit the YAML to your Kubeflow Pipelines instance: kfp.Client().create_run_from_pipeline_package('pipeline.yaml', arguments={})

Adopting v2 gives you early error detection, version‑controlled pipeline definitions, and the ability to run the same definition across multiple platforms. Start by converting a small, self‑contained component to the typed @dsl.component form, compile it, and test locally before moving to the cluster.