From Local Script to Reproducible Workflow: Kubeflow Pipelines v2

The problem: your training script is not a pipeline

Most machine learning teams start with a training script that runs perfectly on a laptop. The same script fails in production because it depends on an undocumented Python version, a library installed globally, or a CSV file sitting in a home directory. There is no record of which parameters produced which model, and no reliable way to rerun the exact experiment.

A useful takeaway: you do not need to rewrite your model logic to fix this. You need a thin orchestration layer that runs your logic inside a pinned container and records every input, output, and metric. Kubeflow Pipelines v2 with the KFP Python SDK gives you that layer by compiling Python workflow definitions into a portable spec that runs on Kubernetes via Argo Workflows, a Kubernetes-native workflow engine.

Keep the pipeline dumb, keep the container smart

The key engineering decision is to keep pipeline orchestration thin and push real logic into tested modules baked into a pinned container image. Components stay portable and the pipeline remains readable rather than embedding training code in the DSL. When a step needs to change, you rebuild the image, not the workflow graph.

This matters because KFP v2 components are not source-compatible with v1. Mixing v1 components or compile targets with v2 pipelines requires checking the migration path. Pinning the base image and validating the generated container commands in the compiled YAML prevents silent drift when decorator options such as base_image or packages_to_install behave differently across SDK versions.

A two-step pipeline in KFP v2

The KFP SDK provides decorators such as @dsl.component and @dsl.pipeline. Inputs, outputs, and metrics are tracked through the metadata store and object storage configured in your Kubeflow Pipelines installation. The example below wraps a preprocess step and a train step as typed components using dsl.Dataset and dsl.Metrics.

from kfp import dsl
from kfp import compiler

@dsl.component(base_image='python:3.11-slim', packages_to_install=['pandas'])
def preprocess(raw_data_uri: str,
               clean_data: dsl.Output[dsl.Dataset]) -> None:
    import pandas as pd
    df = pd.read_csv(raw_data_uri)
    df = df.dropna()
    df.to_csv(clean_data.path, index=False)

@dsl.component(base_image='python:3.11-slim',
               packages_to_install=['pandas', 'scikit-learn'])
def train(clean_data: dsl.Input[dsl.Dataset],
          n_estimators: int,
          model: dsl.Output[dsl.Model],
          metrics: dsl.Output[dsl.Metrics]) -> None:
    import pandas as pd
    from sklearn.ensemble import RandomForestClassifier
    import joblib
    df = pd.read_csv(clean_data.path)
    X = df.drop('target', axis=1)
    y = df['target']
    clf = RandomForestClassifier(n_estimators=n_estimators)
    clf.fit(X, y)
    joblib.dump(clf, model.path)
    metrics.log_metric('n_estimators', n_estimators)

@dsl.pipeline(name='two-step-training')
def training_pipeline(raw_data_uri: str, n_estimators: int = 100):
    preprocess_task = preprocess(raw_data_uri=raw_data_uri)
    train_task = train(clean_data=preprocess_task.outputs['clean_data'],
                       n_estimators=n_estimators)

compiler.Compiler().compile(training_pipeline, 'pipeline.yaml')

Parameters flow through the pipeline function signature, and artifacts pass by reference. The raw_data_uri should point to an object store path reachable from the cluster, not a local file.

Verify locally before you touch the cluster

A local compile does not prove the cluster can run the pipeline, but it does confirm the DSL and compiler are consistent. In a clean virtual environment on your workstation, install the KFP SDK and record its version:

python -m venv kfp-venv
source kfp-venv/bin/activate
pip install kfp==<sdk-version>
python -c "import kfp; print(kfp.__version__)"

Next, compile the pipeline and inspect the generated YAML. A v2 pipeline spec contains top-level keys such as schemaVersion, pipelineInfo, components, root, and deploymentSpec with executor containers. Confirm that parameters and artifact inputs/outputs are wired as intended.

python compile_pipeline.py
grep -E 'schemaVersion|pipelineInfo|deploymentSpec' pipeline.yaml

Running on the cluster and what to check

On a Kubeflow cluster, check the Pipelines UI version and the ml-pipeline deployment image tag before submitting. Run this from a machine with kubeconfig access and appropriate RBAC (role-based access control):

kubectl -n kubeflow get deploy ml-pipeline -o jsonpath='{.spec.template.spec.containers[0].image}'

Upload the compiled YAML through the Pipelines UI and start a run with a tiny input to confirm both steps succeed. After the run, inspect the run page for the logged metrics and the output artifacts to confirm the metadata store and object storage integration are working.

Optionally, port-forward the ml-pipeline service and submit programmatically. Port-forwarding exposes the service on your local machine; treat the endpoint as sensitive.

kubectl -n kubeflow port-forward svc/ml-pipeline 8888:8888

import kfp
client = kfp.Client(host='http://localhost:8888')
client.create_run_from_pipeline_func(
    training_pipeline,
    arguments={'raw_data_uri': 's3://<bucket>/<key>', 'n_estimators': 10}
)

Trade-offs and limitations

• Cluster installation and RBAC complexity. Standing up Kubeflow Pipelines and granting least-privilege access is non-trivial.
• Coupling to Argo Workflows. The backend submits to Argo; debugging sometimes means reading Argo resource definitions.
• Reliance on configured object storage. Missing or misconfigured storage causes runs to fail or lose lineage.
• Version drift. The KFP SDK and the installed backend must match major versions. Confirm the exact Pipelines version and the matching SDK before authoring.
• Artifact size. Large intermediate data passed as artifacts can be slower than shared volumes.

An actionable path

Start small. Containerize one existing script, expose it as a @dsl.component with a pinned base image, compile and run it once, then add parameters and a second component. Verify locally by compiling to YAML before touching the cluster, and check the run page afterward to confirm metrics and artifacts persist.