Why Your CV Score Is Too Good: Preventing Data Leakage with scikit-learn Pipelines

The leakage trap: why preprocessing before CV inflates scores

You split your data into training and validation folds, fit a scaler on the full dataset first, and then run cross_val_score. The accuracy looks excellent. In production, it collapses. The reason is data leakage: your scaler already saw the mean and variance of the validation folds before the model was trained, so the model was evaluated on data it had indirectly memorized.

Data leakage occurs when information from outside the training dataset is used to create the model. In cross-validation, every fold is supposed to simulate unseen data. If you fit a transformer—such as StandardScaler or OneHotEncoder—on the entire dataset before splitting, the training folds absorb statistics from the validation folds. The resulting score is optimistic and not reproducible.

Pipeline and ColumnTransformer: preprocessing that respects the fold boundary

A scikit-learn Pipeline chains multiple steps into a single estimator. When you pass a pipeline to cross_val_score or GridSearchCV, the library clones the pipeline and calls fit only on the training split for each fold. Every transformer inside the pipeline learns its parameters—means, category vocabularies, scaling factors—exclusively from that training split.

Real tabular data rarely has a single type. Numeric columns need scaling; categorical columns need encoding. A ColumnTransformer routes different column subsets to different transformers and concatenates the results. Because the transformer is itself an estimator, it can live inside a pipeline. The composite object exposes the same fit, predict, and score interface as a simple model, so the entire workflow is validated atomically.

Worked example: mixed numeric and categorical features

Suppose you have a pandas DataFrame with two numeric columns and two categorical columns, and a binary target. The goal is to train a logistic regression without leaking validation statistics.

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

# Raw DataFrame; no preprocessing has been applied yet.
# numeric_cols = ["age", "income"]
# categorical_cols = ["city", "plan"]
# target_col = "churn"

numeric_transformer = StandardScaler()
categorical_transformer = OneHotEncoder(handle_unknown="ignore")

preprocessor = ColumnTransformer(
    transformers=[
        ("num", numeric_transformer, numeric_cols),
        ("cat", categorical_transformer, categorical_cols),
    ]
)

clf = Pipeline(
    steps=[
        ("preprocessor", preprocessor),
        ("classifier", LogisticRegression(max_iter=1000)),
    ]
)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(
    clf, df[numeric_cols + categorical_cols], df[target_col], cv=cv, scoring="accuracy"
)
print(scores.mean())

Run this in any Python environment with scikit-learn installed; no special permissions are required. Replace numeric_cols, categorical_cols, and target_col with your actual column names. The expected check is that scores contains one accuracy value per fold. If the pipeline were leaking, you would typically see a higher mean score with lower variance across folds.

To confirm that the scaler respected the fold boundary, fit the pipeline manually on a single training split and inspect the learned statistics:

from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(
    df[numeric_cols + categorical_cols], df[target_col], test_size=0.2, random_state=42
)
clf.fit(X_train, y_train)

# The scaler mean should match X_train, not the full DataFrame.
scaler_mean = clf.named_steps["preprocessor"].named_transformers_["num"].mean_
print(scaler_mean)

If scaler_mean equals the mean of X_train rather than the mean of the full dataset, the pipeline is behaving correctly.

Trade-offs, limitations, and how to verify the fix

Pipelines add conceptual overhead. The intermediate data shape after ColumnTransformer is hidden, which makes debugging harder. When a model underperforms, inspect named_steps or call transform on a small sample to see what the final estimator actually receives.

A common gotcha involves sparse and dense matrices. OneHotEncoder returns a sparse matrix by default, while StandardScaler returns a dense array. When ColumnTransformer concatenates them, the result may be sparse. Some estimators—such as certain SVM or tree implementations—accept sparse input, but others do not. If you encounter an error, set sparse_output=False in OneHotEncoder to force a dense result. Note that the parameter was named sparse before scikit-learn 1.2; verify which version your environment uses.

Schema drift is another limitation. ColumnTransformer selects columns by explicit name or dtype. If production data adds, removes, or reorders columns, the pipeline will raise an error or select the wrong features. For robust deployments, either pin the input schema upstream or use make_column_selector with dtype filters, and re-fit the pipeline when the schema legitimately changes.

Actionable closing

Audit your existing notebooks for any fit_transform call that runs before cross_val_score. Replace that pattern with a Pipeline that contains a ColumnTransformer. Then verify the fix by comparing the mean CV score of the leaked approach against the pipeline approach on the same raw data. The pipeline version should show more conservative and trustworthy scores. Finally, inspect named_steps after a manual fit to confirm that learned statistics come only from the training split.