Building Modular ML Workflows with Kubeflow Pipelines SDK

The Challenge of ML Pipeline Reproducibility

Machine learning workflows often fail when moving from a notebook to production because dependencies differ between the data preprocessing, training, and evaluation stages. Hard-coding these steps into a single script creates a "monolith" that is difficult to debug and impossible to scale across a Kubernetes cluster.

The solution is to decompose the workflow into Components—independent, containerized units of execution. By using the Kubeflow Pipelines (KFP) SDK, you can define these components as Python functions, compile them into a Directed Acyclic Graph (DAG), and execute them via the Argo Workflow engine.

Prerequisites

• A running Kubeflow cluster (v1.7+ recommended).
• Python 3.8+ environment with kfp installed.
• Access to an S3-compatible object store (MinIO is the default in Kubeflow) for artifact storage.
• kubectl configured to access the cluster for log verification.

Step 1: Defining Lightweight Components

The most efficient way to build components is using the @component decorator. This allows the SDK to automatically generate the Docker image requirements and the component specification without requiring a manual Dockerfile for every small function.

from kfp import dsl

@dsl.component(
    base_image="python:3.9",
    packages_to_install=["pandas", "scikit-learn"]
)
def preprocess_data(input_csv: str, output_dataset: dsl.Output[dsl.Dataset]):
    import pandas as pd
    # Load data from a URI
    df = pd.read_csv(input_csv)
    # Simple cleaning logic
    df_cleaned = df.dropna()
    # Write to the KFP-managed artifact path
    df_cleaned.to_csv(output_dataset.path, index=False)

Step 2: Orchestrating the Pipeline DAG

Once components are defined, you must wrap them in a @pipeline function. This is where you define the data flow. In KFP, passing the output of one component as the input to another creates a dependency edge in the DAG.

@dsl.pipeline(name="ML-Training-Pipeline")
def training_pipeline(csv_url: str):
    # Step 1: Preprocess
    prep_task = preprocess_data(input_csv=csv_url)
    
    # Step 2: Train (Assuming train_model is another @component)
    # The output_dataset from prep_task is passed here
    train_task = train_model(dataset=prep_task.outputs['output_dataset'])
    
    # Ensure training only starts after preprocessing completes
    train_task.after(prep_task)

Step 3: Compilation and Deployment

The pipeline must be compiled into a YAML specification before it can be uploaded to the Kubeflow UI. Run this command in your local Python environment:

from kfp import compiler

compiler.Compiler().compile(pipeline_func=training_pipeline, package_path="pipeline.yaml")

Risk: If you use a base_image that is incompatible with the packages_to_install, the pod will enter a CrashLoopBackOff state. Always match your Python version to the libraries you intend to use.

Comparison: Parameters vs. Artifacts

Choosing the wrong data passing method can lead to performance bottlenecks or pipeline failures.

Feature	Parameters (Strings/Ints)	Artifacts (Datasets/Models)
Storage	Stored in ML Metadata (MLMD) database	Stored in S3/MinIO object store
Size Limit	Very small (kilobytes)	Very large (gigabytes/terabytes)
Use Case	Hyperparameters, file paths, flags	CSV files, TF SavedModels, NumPy arrays

Verification and Diagnostics

After uploading the pipeline.yaml via the Kubeflow Dashboard and triggering a run, use these checks to verify the state:

1. UI Graph Check: Ensure the DAG shows a linear progression from preprocess-data to train-model. If the components are not linked, the SDK failed to recognize the output-to-input dependency.
2. Pod Log Inspection: If a step fails, find the pod name in the namespace and run:

   kubectl logs -n kubeflow-user-example-com [POD_NAME]

3. Artifact Validation: Check the "Artifacts" tab in the Kubeflow UI to ensure the output_dataset URI exists and contains data.

Limitations and Constraints

• Static DAGs: You cannot use standard Python for loops to create dynamic steps based on runtime data. You must use dsl.ParallelFor for iterations.
• Disk Pressure: When handling massive datasets, avoid reading the entire file into memory. Use streaming or chunking to prevent the Kubernetes node from hitting OOM (Out of Memory) limits.

Rollback Procedure

Because KFP pipelines are immutable once compiled, you cannot "edit" a running pipeline. To roll back a change:

1. Revert the Python code to the previous stable commit.
2. Re-compile the pipeline.yaml.
3. Upload the YAML as a new version in the Kubeflow UI.
4. Delete the failing run to clear any locked resources in the object store.