Compile and Run a Kubeflow Pipelines v2 Pipeline with the KFP SDK v2

The problem and what this guide covers

You have a Kubeflow cluster with Kubeflow Pipelines installed, but you need to author a pipeline using the KFP SDK v2, compile it to the v2 intermediate representation, submit it to the cluster, and confirm that the run succeeded. Kubeflow Pipelines v2 uses the KFP SDK v2 to define components as Python functions decorated with @component and pipelines with @pipeline, then compiles them to a YAML file that the Pipelines API can execute.

This guide walks through a minimal end-to-end example: write the pipeline, compile it locally, authenticate to an existing Kubeflow Pipelines endpoint, submit the run, and verify it using both the SDK and kubectl. It assumes you already have a running Kubeflow deployment; it does not cover installing Kubeflow or Pipelines.

Prerequisites

• A running Kubeflow cluster with the Kubeflow Pipelines API service exposed and reachable from your workstation.
• Python 3.8 or later with the kfp SDK v2 installed on your local workstation.
• Network access to the Pipelines endpoint and authentication configured for your cluster (Dex, OIDC, or session cookies, depending on the deployment).
• Container images referenced by your components must be pullable by the cluster nodes; private registries require imagePullSecrets in the target namespace.

Author a minimal v2 pipeline

Create a file named pipeline.py on your local workstation. The following example defines two components and a pipeline that runs them sequentially.

from kfp import dsl, component
from kfp.dsl import pipeline

@component
def add_numbers(a: float, b: float) -> float:
    '''Add two numbers and return the result.'''
    return a + b

@component
def multiply_by_factor(value: float, factor: float) -> float:
    '''Multiply a value by a factor.'''
    return value * factor

@pipeline(
    name='minimal-v2-pipeline',
    description='A minimal Kubeflow Pipelines v2 pipeline.'
)
def minimal_pipeline(a: float = 2.0, b: float = 3.0, factor: float = 10.0):
    add_task = add_numbers(a=a, b=b)
    multiply_task = multiply_by_factor(value=add_task.output, factor=factor)

Run this file locally with Python to confirm it imports without syntax errors. Successful import does not mean the pipeline will run on the cluster; it only validates the Python code.

Compile the pipeline to YAML

On your local workstation, run the KFP v2 compiler to produce the pipeline package:

python -m kfp.compiler.compile --py pipeline.py --output pipeline.yaml

Alternatively, in Python:

from kfp.compiler import Compiler
Compiler().compile(pipeline_func=minimal_pipeline, package_path='pipeline.yaml')

The compiler produces pipeline.yaml, the v2 intermediate representation. Compilation succeeds locally even if container images fail to pull at runtime, so image availability must be validated separately.

Submit the pipeline to Kubeflow

On your local workstation, create a Python script to authenticate and submit the run. Authentication is cluster-specific; do not assume bearer tokens work without configuration.

from kfp import Client

# Replace with your Kubeflow Pipelines endpoint
host = 'https://pipelines.your-cluster.example.com'

client = Client(host=host)

run = client.create_run_from_pipeline_package(
    pipeline_package_path='pipeline.yaml',
    arguments={'a': 2.0, 'b': 3.0, 'factor': 10.0},
)

print(run.run_id)

Where to run: your local workstation with network access to the cluster. Permissions: your user account must have permission to create runs in the target Kubeflow namespace. Expected check: the script prints a run ID. Risk: authentication fails if your cluster uses Dex or OIDC and you have not completed the login flow or configured session cookies.

Verify the run

After submission, confirm the run through the Kubeflow Pipelines UI and through kubectl.

In the Pipelines UI, locate the run by ID and confirm it transitions through Pending, Running, and Succeeded.

From a terminal with access to the cluster, run:

kubectl get pods -n kubeflow -l pipelines.kubeflow.org/run-id=<run-id>

Where to run: a machine with kubectl configured for the cluster. Permissions: read access to pods in the kubeflow namespace, or the namespace where Pipelines runs workloads. Expected check: workload pods appear and eventually reach Completed or Error status.

To inspect a failed step:

kubectl logs -n kubeflow <pod-name>

Common failures and recovery

Symptom	Likely cause	Recovery
Run stays in Pending	Cluster resources unavailable or image cannot be pulled	Check pod events with kubectl describe pod; verify imagePullSecrets for private registries
Pod fails with ImagePullBackOff	Container image not accessible from cluster nodes	Push image to a registry reachable by the cluster or configure imagePullSecrets
Authentication error from kfp.Client	Missing or expired session token for Dex/OIDC	Complete the cluster-specific login flow before creating the Client
Component step fails	Component code error or missing input artifact	Fix the Python component, recompile, and submit a new run

Submitting a pipeline creates a run in the cluster. There is no in-place edit of a submitted run; to change behavior, fix the component code or pipeline arguments, recompile, and create a new run. You can archive or delete the old run through the Pipelines UI if it is no longer needed.

Limitations and pre-flight checks

• Local compilation does not validate that container images exist or are pullable by cluster nodes. Validate image availability separately before relying on a pipeline run.
• Namespaced resources such as PersistentVolumeClaims must exist in the same namespace as the pipeline run or be declared in the component.
• Authentication depends on your Kubeflow deployment; this guide does not provide a universal token method.
• This guide targets Kubeflow Pipelines v2 with the KFP SDK v2. Behavior may differ with SDK v1.

Practical verification: after the run completes, open the run details page in the Kubeflow Pipelines UI and confirm that output artifacts and metrics are recorded for each successful step. If artifacts are missing, inspect the step logs and the ML Metadata store for the run.