When Kubeflow Pipelines Earns Its Place (and When a CronJob Wins)

Your model retraining script works fine as a Kubernetes CronJob until someone asks: which version of the data produced last Tuesday's model, and why did accuracy drop? That question — reproducibility plus lineage — is exactly where Kubeflow Pipelines (KFP) starts paying for its operational cost. This post is about making that call honestly, with a concrete example of what KFP gives you.

The problem KFP actually solves

KFP lets you define an ML workflow in Python: each step is a containerized component, and the pipeline wires typed artifacts (datasets, models, metrics) between them. The SDK compiles your pipeline to a YAML intermediate representation that runs on Kubernetes.

Three capabilities matter most in practice:

• Step-level caching. If a component's inputs and container image haven't changed, KFP reuses its previous output instead of re-executing. Rerunning a pipeline after tweaking only the training step skips an hour of feature engineering.
• Lineage via ML Metadata (MLMD). Every run records which data, code, and parameters produced which model. That's the audit trail the CronJob can't give you.
• Scheduling and a UI. Recurring runs, run comparison, and artifact browsing out of the box.

A minimal two-step pipeline (KFP v2 SDK)

The v2 SDK is decorator-based — note that v1 code using ContainerOp looks completely different, so check which major version your cluster supports before copying examples. Run this locally with the KFP SDK installed (pip install kfp, v2.x); it compiles without needing a cluster:

from kfp import dsl
from kfp.dsl import component, Dataset, Model, Metrics

@component(base_image="python:3.11-slim")
def preprocess(raw_path: str, output_dataset: Output[Dataset]):
    import pandas as pd
    df = pd.read_csv(raw_path).dropna()
    df.to_parquet(output_dataset.path)

@component(base_image="python:3.11-slim")
def train(dataset: Input[Dataset], model: Output[Model], metrics: Output[Metrics]):
    import pandas as pd, pickle
    from sklearn.linear_model import LogisticRegression
    df = pd.read_parquet(dataset.path)
    clf = LogisticRegression().fit(df.drop("y", axis=1), df["y"])
    metrics.log_metric("accuracy", float(clf.score(df.drop("y", axis=1), df["y"])))
    with open(model.path, "wb") as f:
        pickle.dump(clf, f)

@dsl.pipeline(name="retrain-pipeline")
def retrain(raw_path: str):
    prep = preprocess(raw_path=raw_path)
    train(dataset=prep.outputs["output_dataset"])

Compile with kfp.compiler.Compiler().compile(retrain, "retrain.yaml") and inspect the YAML — you'll see the two components and the artifact wiring. Upload it through the KFP UI or the SDK client, run it twice with the same raw_path, and the second run should show the preprocess step as cached (verify in the run graph; cached steps are marked rather than re-executed).

The decision framework

Adopt KFP when most of these hold: you already run Kubernetes, multiple people touch the pipeline, retraining is recurring, and you need to answer "what produced this model?" months later. Standalone KFP installation (without the full Kubeflow platform's notebooks, Katib, and KServe) keeps the operational footprint reasonable.

Stick with a CronJob or a lighter orchestrator when it's one script, one owner, and the inputs are self-describing. KFP adds a control plane, a metadata database, and object storage for artifacts — real things that break at 2 a.m.

Trade-offs worth knowing upfront

• Caching can hide bugs. The cache key covers code image and declared inputs. If your component reads an external table that mutated in place, KFP may serve stale outputs. Make inputs explicit parameters, not ambient side effects.
• Artifacts flow through object storage. Every dataset passed between steps is serialized there. For very large intermediates, consider passing references (URIs) instead of data, or use shared volumes.
• Version compatibility moves. SDK major versions, Kubernetes support, and manifest compatibility change between releases — check the official KFP release notes before pinning versions. Don't assume from memory, including this post.

Your next step

Compile the example above locally and read the generated YAML — that costs nothing and tells you whether the model fits your head. If it does, deploy standalone KFP on a test cluster, run the pipeline twice, and confirm caching and lineage in the UI before committing your team to it.