Stop Leaking Your Test Set: ColumnTransformer + Pipeline for Mixed Data in scikit-learn

Most real datasets are messy: a few numeric columns like age and salary, a few categorical ones like gender and country. The classic mistake is preprocessing them separately—fit a StandardScaler here, a OneHotEncoder there, concatenate by hand—and accidentally fitting those transformers on the entire dataset before splitting. That quietly leaks test-set statistics into training, inflating your validation scores with numbers you won't see in production.

The fix is structural, not vigilance-based: wrap a ColumnTransformer inside a scikit-learn Pipeline. The pipeline enforces a strict fit/transform sequence, so every statistic (means, category lists) is learned only from the training split and then applied unchanged to validation and test data.

Why manual preprocessing leaks

Data leakage happens when information from the test set influences training. If you call scaler.fit(X) before train_test_split, the scaler's mean and standard deviation include test rows. Your model is then evaluated on data that partially shaped its own features—scores look better than reality.

Manual per-column preprocessing also produces verbose, error-prone code: separate fitted objects to track, separate transform calls, and no guarantee the same steps run in the same order at inference time.

ColumnTransformer: one transformer, per-column rules

ColumnTransformer (available since scikit-learn 0.20) applies different transformers to different column subsets and concatenates the results. Each entry is a tuple of (name, transformer, columns). Columns can be names (for DataFrames) or integer positions.

Embedding it in a Pipeline means pipeline.fit(X_train, y_train) fits the preprocessing and the model on training data only. pipeline.predict(X_test) then reuses those frozen transformations. Cross-validation via cross_val_score also becomes safe, because the pipeline is re-cloned and re-fit inside each fold.

Worked example: numeric + categorical to logistic regression

Run this in any Python environment with scikit-learn 1.2+ installed (check with python -c "import sklearn; print(sklearn.__version__)"). No special permissions needed; it uses synthetic data so you can paste it directly.

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
df = pd.DataFrame({
    "age": rng.integers(18, 70, 500),
    "gender": rng.choice(["f", "m", "nb"], 500),
})
y = (df["age"] > 40).astype(int)

numeric_pipe = Pipeline([("scaler", StandardScaler())])
categorical_pipe = Pipeline([("onehot", OneHotEncoder(handle_unknown="ignore"))])

preprocess = ColumnTransformer([
    ("num", numeric_pipe, ["age"]),
    ("cat", categorical_pipe, ["gender"]),
])

model = Pipeline([
    ("prep", preprocess),
    ("clf", LogisticRegression()),
])

X_train, X_test, y_train, y_test = train_test_split(
    df, y, test_size=0.25, random_state=0
)
model.fit(X_train, y_train)
print(model.score(X_test, y_test))

Two details matter here. First, handle_unknown="ignore" on the encoder means a category seen only in test data (or production) is encoded as all zeros instead of raising an error. Second, the scaler's mean is computed from X_train only. You can verify the no-leakage property: inspect model.named_steps["prep"].named_transformers_["num"].named_steps["scaler"].mean_ and compare it to X_train["age"].mean()—they should match exactly, and differ from the full-data mean.

Another quick check: transform X_train and X_test via model.named_steps["prep"].transform(...) and confirm both outputs have the same number of columns, so the model sees a consistent feature space.

Trade-offs and limitations

Debugging opacity. A pipeline hides intermediate state. When a step fails, you must dig in via pipeline.named_steps or call transform on the preprocessing step alone to inspect arrays. Custom transformers must implement the fit(self, X, y) / transform(self, X) API exactly (subclass BaseEstimator and TransformerMixin to get this right).

Memory with high-cardinality categoricals. By default, recent scikit-learn versions return sparse output from OneHotEncoder, but if you densify it (or use an older version with sparse=False), a column with thousands of unique values can explode memory. Keep sparse output and use a solver that accepts it (LogisticRegression with solver="saga" or liblinear), or consider alternative encodings for very high-cardinality features.

Version and reproducibility notes. ColumnTransformer behavior around remainder="passthrough" and column ordering shifted between 0.20 and 0.24—if you see unexpected column order, check your version and the release notes. Also, pipelines cloned during cross-validation don't magically seed stochastic estimators; always set random_state explicitly on components like SGDClassifier.

Actionable closing

If your preprocessing lives outside a Pipeline, assume it leaks until proven otherwise. Move every fitted transformer into a ColumnTransformer, place it before your estimator, and fit only on training data. Then verify: compare the scaler's stored mean_ against the training-split mean, and confirm train/test transformed shapes match. Those two checks take a minute and catch the most common leakage bug in tabular ML workflows.