Choosing Between Kubeflow Pipelines v1 and v2 SDKs for New ML Projects

The Decision: v1 vs v2 SDK

When starting a new machine learning pipeline, you must choose between the legacy Kubeflow Pipelines (KFP) v1 SDK (versions < 2.0) and the v2 SDK (versions >= 2.0). This choice dictates how you define components, how the pipeline is compiled into YAML, and how artifacts are tracked in the Kubeflow UI.

The primary constraint is your Kubeflow control plane version. While newer versions of the ml-pipeline backend support both, the v2 SDK is the current development path and provides significant improvements in reproducibility and metadata tracking.

Comparison of SDK Capabilities

Feature	KFP v1 SDK	KFP v2 SDK
Development Status	Maintenance mode	Active development
Compiler Output	Argo Workflow (legacy spec)	Argo Workflow (v2 spec)
Component Definition	ContainerOp / @dsl.pipeline	@dsl.component / ComponentSpec
Artifact Handling	Implicit/Manual path passing	Strongly typed Input/Output artifacts
UI Experience	Basic run history	Enhanced artifact lineage visualization
Portability	Dependent on shared base images	Hermetic components (self-contained)

Engineering Trade-offs

Choose the v2 SDK if:

• You are using Kubeflow 1.8+ or a managed service that supports KFP v2.
• You require strict reproducibility (Hermetic components ensure the environment is packaged with the component).
• You need to visualize the flow of data (artifacts) between steps in the UI.
• You want better IDE support through Python type hints for pipeline inputs and outputs.

Stick with the v1 SDK if:

• You have a massive library of existing v1 custom components that would be too costly to rewrite.
• Your environment is locked to an older Kubeflow distribution (pre-1.8) that does not support the v2 metadata writer.
• You rely on specific legacy integrations (like certain Katib TrialTemplate configurations) that have not yet been migrated to v2.

Implementation Example: KFP v2 Pipeline

The v2 SDK moves away from manual string-based path passing and uses Input and Output types to manage artifacts. This allows the KFP backend to handle the actual storage locations automatically.

1. Define the Pipeline

import kfp
from kfp import dsl
from kfp.dsl import component, Input, Output, Artifact

@component
def preprocess_data(input_text: str, output_artifact: Output[Artifact]) -> None:
    # In v2, we write to the path provided by the output_artifact object
    with open(output_artifact.path, 'w') as f:
        f.write(f"Processed: {input_text}")

@component
def train_model(input_artifact: Input[Artifact], model_output: Output[Artifact]) -> None:
    with open(input_artifact.path, 'r') as f:
        data = f.read()
    
    with open(model_output.path, 'w') as f:
        f.write(f"Model trained on {data}")

@dsl.pipeline(name='v2-decision-demo')
def ml_pipeline(text_input: str = "default_data"):
    prep_task = preprocess_data(input_text=text_input)
    # The output of prep_task is automatically passed as an Input artifact to train_task
    train_task = train_model(input_artifact=prep_task.outputs['output_artifact'])

if __name__ == '__main__':
    kfp.compiler.Compiler().compile(ml_pipeline, 'pipeline.yaml')

2. Compilation and Validation

Run the script in a Python environment with kfp>=2.0. To verify the compilation is targeting v2, inspect the pipeline.yaml file. You should see apiVersion: argoproj.io/v1alpha1 and a structure that reflects the ComponentSpec rather than legacy ContainerOp definitions.

3. Execution and Verification

Run the following commands to upload and execute the pipeline. This requires a kfp.Client connected to a cluster where you have permissions to create runs in your assigned namespace.

# Using the KFP Client to execute
client = kfp.Client(host='https://your-kubeflow-endpoint')

# Create a run from the function
run = client.create_run_from_pipeline_func(
    ml_pipeline,
    arguments={'text_input': 'sample_dataset_v1'},
    experiment_name='v2-test-experiment'
)

print(f"Run created: {run.run_id}")

Verification Check: Open the Kubeflow UI, navigate to the run, and check the Artifacts tab. If the v2 metadata writer is working, you will see the output_artifact and model_output listed as distinct entities with a lineage graph connecting them.

Operational Limitations

• SDK Versioning: Minor versions of the v2 SDK (e.g., 2.0 vs 2.1) have introduced breaking changes in the compiler output. It is critical to pin your SDK version in requirements.txt (e.g., kfp==2.x.x) to prevent pipeline compilation failures during CI/CD.
• Control Plane Version: If you see errors regarding metadata writing or the artifacts tab is empty, verify your ml-pipeline image version:

  kubectl get deployment ml-pipeline -n kubeflow -o yaml | grep image

  If the version is older than 1.8, v2 features will be partially unsupported.
• TFX Compatibility: If using TensorFlow Extended (TFX), ensure you are using TFX 1.13+ to leverage v2-compatible component APIs.

Rollback

Since this operation involves creating runs on a cluster, you can remove a failed test run via the client to clean up the namespace:

client.delete_run(run_id='your-run-id')

Takeaway

For any new project on Kubeflow 1.8+, the v2 SDK is the recommended choice due to its superior artifact tracking and hermetic component model. Only opt for v1 if you are constrained by a legacy control plane or a large existing library of v1 components that cannot be migrated.