Breaking the Monolith: Moving to Modular Kubeflow Pipelines

The 'One Big Script' Bottleneck

Many machine learning projects start as a single, massive Python script. It handles everything: downloading data, cleaning it, training the model, and saving the weights. While this works for a prototype, it fails in production. If the training step crashes after four hours, you cannot simply restart from the failure point; you have to rerun the entire ingestion and preprocessing sequence.

The solution is to decompose the workflow into a Directed Acyclic Graph (DAG)—a series of independent, containerized steps where each output serves as the input for the next. In Kubeflow Pipelines (KFP), this modularity allows you to isolate failures, reuse components across different projects, and save compute costs through caching.

Modularizing with Lightweight Python Components

Historically, creating a pipeline component required writing a Dockerfile, building an image, and pushing it to a registry. KFP simplifies this via Lightweight Python Components (LPC). By using the @component decorator, you define the function's requirements, and KFP handles the containerization at runtime.

This approach separates the logic of the step from the infrastructure of the cluster. You define the base image (e.g., a standard Python image) and the packages needed, and KFP ensures that the specific environment is instantiated for that single task.

Worked Example: A Two-Step Preprocessing Pipeline

To implement this, you need the kfp SDK installed in your local development environment. Run pip install kfp to get started. The following example demonstrates how to split a data-loading step from a processing step.

from kfp import dsl
from kfp import compiler

# Component 1: Data Ingestion
@dsl.component(base_image='python:3.9', packages_to_install=['pandas'])
def ingest_data(input_url: str, output_dataset: dsl.Output[dsl.Dataset]):
    import pandas as pd
    df = pd.read_csv(input_url)
    # Logic to filter or clean raw data
    df.to_csv(output_dataset.path, index=False)

# Component 2: Data Processing
@dsl.component(base_image='python:3.9', packages_to_install=['pandas', 'scikit-learn'])
def process_data(input_dataset: dsl.Input[dsl.Dataset], output_metrics: dsl.Output[dsl.Metrics]):
    import pandas as pd
    df = pd.read_csv(input_dataset.path)
    # Example: calculate a simple metric
    row_count = len(df)
    output_metrics.log_metric("row_count", row_count)

# Define the Pipeline DAG
@dsl.pipeline(name='modular-ml-pipeline')
def ml_pipeline(url: str = 'https://example.com/data.csv'):
    ingest_task = ingest_data(input_url=url)
    process_task = process_data(input_dataset=ingest_task.outputs['output_dataset'])

# Compile to YAML for deployment
compiler.Compiler().compile(ml_pipeline, 'pipeline.yaml')

Execution and Verification

• Run Location: Execute the compilation script locally. Upload the resulting pipeline.yaml to the Kubeflow UI.
• Permissions: Ensure your Kubernetes service account has write access to the default artifact store (usually MinIO or S3).
• Verification: In the Kubeflow DAG visualization, verify that process_data has a dependency arrow pointing from ingest_data.

The Trade-off: Caching vs. Data Freshness

KFP uses caching to avoid redundant work. If you run a pipeline with the same parameters and the input components haven't changed, KFP skips those steps and uses the previous output. This is a massive time-saver during hyperparameter tuning.

However, there is a critical limitation: KFP is unaware of external data changes. If your input_url points to a CSV file that is updated daily on a server, KFP will see the URL string is the same and skip the ingest_data step, serving you stale data from the previous run. To fix this, include a version timestamp or a hash of the data as a pipeline parameter to force a cache miss when the data changes.

Handling Large Datasets

A common mistake is passing large DataFrames directly as function arguments. This forces the data to be serialized and passed through the KFP metadata store, which creates significant I/O overhead and can crash the pipeline.

Instead, use dsl.Input[dsl.Dataset] and dsl.Output[dsl.Dataset]. These objects act as pointers to files in object storage. The component writes the file to a path, and the next component reads from that same path, keeping the heavy data off the KFP control plane.

Actionable Summary

To move from a script to a modular pipeline: first, identify the natural boundaries of your workflow (Ingestion → Validation → Training → Eval). Wrap these in @component functions using the KFP SDK. Use dsl.Dataset for large data transfers to avoid memory bottlenecks. Finally, always parameterize your data versions to ensure the caching mechanism doesn't accidentally serve you outdated results.