Kubeflow Pipelines SDK v2: Compile, Submit, and Avoid Common Pitfalls

The Problem and the Takeaway

Teams adopting Kubeflow Pipelines often struggle with version mismatches between the kfp SDK and the deployed controller, stale caching that skips necessary re-runs, and ad-hoc notebook submissions that drift from CI/CD discipline. The practical engineering decision: compile pipelines in CI, store the resulting Argo YAML as an artifact, and submit that artifact with run‑specific parameters and secrets. This avoids rebuilding component images on every change and keeps the deployment reproducible.

How the SDK v2 Compilation Works

Kubeflow Pipelines SDK v2 uses Python decorators (@dsl.component and @dsl.pipeline) to define components and pipelines. The Compiler emits an Argo‑flavored YAML package that the KFP service schedules as Kubernetes jobs. The YAML contains workflow templates and a container template referencing the component’s base image.

Worked Example: A Minimal Add Pipeline

import kfp
from kfp import dsl
from kfp.compiler import Compiler

@dsl.component(base_image='python:3.9')
def add(a: float, b: float) -> float:
    return a + b

@dsl.pipeline(name='add-pipeline', description='adds two numbers')
def pipeline(a: float = 1.0, b: float = 2.0):
    add(a=a, b=b)

Compiler().compile(pipeline_func=pipeline, package_path='add.yaml')

Run this script on a development machine or in a CI job where the kfp SDK v2 is installed. The output add.yaml is the portable pipeline package.

Submitting the Compiled Package

import kfp

client = kfp.Client(host='https://kubeflow.example.com/pipeline')
run = client.create_run_from_pipeline_package(
    package_path='add.yaml',
    arguments={'a': 3, 'b': 4},
    enable_caching=False  # disable when code changes without input changes
)
print(f'Run ID: {run.run_id}')

Where to run: Any environment with network access to the Kubeflow Pipelines API endpoint (typically a developer workstation, CI runner, or a cluster pod with a service account).
Permissions: The client authenticates via the ambient Kubernetes credentials (service account token, kubeconfig, or OIDC). The service account must have create permission on workflows.argoproj.io in the target namespace.
Placeholders: Replace https://kubeflow.example.com/pipeline with your KFP API host. Adjust arguments per run.
Expected checks: Poll client.get_run(run_id) or watch the UI until the run reaches Succeeded. Verify the output artifact (the sum) appears in the run’s artifact list.
Risks: If enable_caching=True (default) and the component implementation changes without a signature change, the previous cached output will be reused silently.

Limits and Common Mistakes

Version Sensitivity

The SDK v2 DSL is not interchangeable with v1. Align the SDK minor version (e.g., 2.8.x) with the deployed Kubeflow Pipelines controller. Newer compiler features (like dsl.Condition enhancements) may produce YAML that older controllers reject. Before adopting new DSL features, compare the installed kfp minor version against the target Kubeflow release notes.

Caching Behavior

KFP caches runs based on component signature (code hash) and input values. Changing the component’s internal logic without changing its signature or inputs leads to stale outputs. Mitigations:

• Set enable_caching=False per run (as shown above) or per component via @dsl.component(..., enable_caching=False).
• Bump a dummy input parameter (e.g., a cache_buster string) when the implementation changes.

Secrets and Credentials

Never hard‑code credentials, service account tokens, or API keys in component source. Pass them as pipeline parameters bound to Kubernetes secrets, or use the platform’s secret injection mechanism (e.g., kfp.dsl.Secret or the Kubeflow UI secret manager). The compiled YAML will reference the secret by name, not value.

Component Base Images

Every @dsl.component must specify a base_image that is pullable by the cluster and contains all imported Python packages. If the image is missing a dependency, the pod fails with ImagePullBackOff or ModuleNotFoundError. Build and push custom images in CI; reference them by digest (e.g., my-registry/component:v1.2.3@sha256:…) for immutability.

Artifact Handling

For outputs larger than a few kilobytes (models, datasets, pandas DataFrames), write files to an OutputPath or a shared volume and return artifact metadata. Do not return large in‑memory objects through the component’s Python return value; the SDK serializes return values as JSON and stores them in the workflow’s parameter store, which has size limits.

Infrastructure Prerequisites

The SDK cannot create namespaces, RBAC roles, or object‑storage buckets. Those resources must exist in the Kubeflow deployment before runs can read/write artifacts. Ensure the target namespace has a configured artifact bucket (e.g., MinIO, S3, GCS) and that the pipeline’s service account has read/write access.

Verification Checklist

1. Confirm SDK v2 is importable: python -c "import kfp; print(kfp.__version__)" – output should start with 2..
2. Compile the example locally and inspect add.yaml for workflowSpec and a container template referencing python:3.9.
3. Submit add.yaml to a running KFP endpoint, poll client.get_run(run_id) until status.condition == 'Succeeded'.
4. Verify the output artifact (the sum) is readable via the UI or client.get_run_artifacts(run_id).

Rollback Consideration

Submitting a pipeline run creates a new Argo Workflow resource. If a bad run is submitted, delete the workflow (kubectl delete workflow -n ) or use the UI’s “Delete” action. No schema migration or stateful rollback is needed because each run is independent.

Summary

Compile once in CI, store the YAML, submit many times with different parameters. Keep SDK and controller versions in lockstep, disable caching when logic changes, inject secrets via Kubernetes, and ensure base images and RBAC are ready before the first run. This pattern yields reproducible, auditable pipeline executions on Kubeflow Pipelines.