Reusable Data Preprocessing with Kubeflow Pipelines

Problem: Code Drift in Preprocessing

When data scientists prototype locally, it’s tempting to copy‑paste a preprocessing function into every notebook. In a production Kubernetes environment this leads to:

• Inconsistent scaling parameters across experiments.
• Hard‑to‑reproduce results when moving code to a cluster.
• Manual effort to rebuild Docker images for every change.

Re‑implementing the same logic in every notebook also makes it difficult to enforce CI/CD pipelines and version control.

Thesis: Encapsulate with a Kubeflow Pipelines Component

A component is a self‑contained unit of work that runs in its own container. By wrapping a preprocessing step in a component you get:

• Versioned, shareable code.
• Explicit input/output contracts.
• Automatic artifact handling via Kubeflow Pipelines’ artifact store.

Below is a concrete example that reads a CSV from a PersistentVolumeClaim, scales numeric columns with scikit-learn, and writes the transformed data back.

1. Define the Component with the @component Decorator

Install the SDK locally:

pip install kfp==2.8.0

Now create preprocess.py:

from kfp.v2.dsl import component, Input, Output, Dataset, Model
import pandas as pd
from sklearn.preprocessing import StandardScaler

@component(
    base_image="python:3.10-slim",
    packages_to_install=["pandas", "scikit-learn"],
    output_component_file="preprocess_component.yaml"
)

def preprocess(
    input_csv: Input[Dataset],
    output_csv: Output[Dataset]
):
    # Load data
    df = pd.read_csv(input_csv.path)
    # Scale numeric columns
    scaler = StandardScaler()
    scaled = scaler.fit_transform(df.select_dtypes(include=["number"]))
    df.loc[:, df.select_dtypes(include=["number"]).columns] = scaled
    # Write back
    df.to_csv(output_csv.path, index=False)

Running the decorator will automatically build a Docker image (using the specified base image and packages), push it to the default registry, and produce a YAML component spec.

2. Assemble a Simple Pipeline

Create pipeline.py that calls the component:

from kfp.v2 import dsl
from kfp.v2.dsl import Dataset

# Import the generated component spec
preprocess = dsl.Component.from_yaml("preprocess_component.yaml")

@dsl.pipeline(name="preprocess-demo")
def preprocess_pipeline(input_path: str, output_path: str):
    # Read input from a PVC
    read_csv = dsl.ContainerOp(
        name="read-csv",
        image="python:3.10-slim",
        command=["bash", "-c"],
        arguments=["cat {input_path} > /tmp/input.csv"],
        file_outputs={"path": "/tmp/input.csv"}
    )
    # Call the preprocessing component
    preprocess_step = preprocess(
        input_csv=read_csv.outputs["path"],
        output_csv=output_path
    )
    # Write output to PVC
    write_csv = dsl.ContainerOp(
        name="write-csv",
        image="python:3.10-slim",
        command=["bash", "-c"],
        arguments=["cat {output_path} > {output_path}"],
        file_outputs={"path": output_path}
    )

Note: In a real cluster you would use Dataset from a PVC or GCS bucket instead of temporary files. The example keeps it simple for local testing.

3. Compile to Argo Workflow YAML

Compile the pipeline to a YAML file that can be applied to a Kubeflow cluster:

from kfp.v2 import compiler
compiler.Compiler().compile(
    pipeline_func=preprocess_pipeline,
    package_path="preprocess_pipeline.yaml"
)

The resulting preprocess_pipeline.yaml contains an Argo workflow definition that Kubeflow Pipelines can ingest.

4. Deploy and Verify

1. Apply the YAML to the cluster:
   

   kubectl apply -f preprocess_pipeline.yaml
   

2. Open the Kubeflow Pipelines UI and launch the preprocess-demo pipeline.
3. Check the logs of the preprocessing pod; you should see the scaling summary and the output file written to the PVC.
4. Verify the artifact in the UI’s Artifacts tab matches the expected scaled CSV.

If the pod fails to start, common culprits are:

• Image not pushed to a registry the cluster can pull from.
• Missing Dataset PVC mount paths.
• Mismatched SDK and cluster version (check compatibility matrix).

Trade‑Offs & Limitations

• Each component requires a Docker build. For large teams the registry bandwidth can become a bottleneck.
• The SDK adds a thin abstraction; you still need to understand container basics and Kubernetes networking.
• Component versions are tied to image tags. Rolling back to an older preprocessing logic requires pulling the correct image.

Despite these overheads, the reproducibility gains and CI/CD friendliness usually outweigh the costs, especially when scaling to dozens of experiments.

Actionable Takeaway

Start by turning a single, frequently used preprocessing function into a component. Test it locally with kfp dsl compile, run it on a test cluster, and then add it to your CI pipeline. Once you’re comfortable, expand the component set to include feature engineering, model training, and evaluation steps. Over time you’ll build a library of vetted, versioned components that make your ML workflows reliable and portable.