Choosing a Missing-Value Strategy for scikit-learn Pipelines

Decision: Handling Missing Values in a Pipeline

When you build a scikit-learn pipeline for tabular data, you must decide how to treat missing values (represented as np.nan) without letting information from validation folds influence training. The decision hinges on available compute time, the degree of correlation between features, and whether you are willing to restrict yourself to a specific estimator family.

Comparison of Options

Approach	Pipeline Integration	Compute Cost	Model Compatibility	Primary Trade‑off
SimpleImputer	Stateless per‑feature	Very Low	Any estimator	Fast but assumes missing at random and can distort distributions.
IterativeImputer	Models each feature from others	Moderate to High	Any estimator	Captures multivariate relationships but may overfit small samples and needs a fixed random_state.
Native handling (HistGradientBoosting)	No imputer required	Low	HistGradientBoostingClassifier/Regressor only	Eliminates imputation bias but locks you into the tree‑based gradient‑boosting family.

Trade‑offs

SimpleImputer

Using mean, median, or most_frequent strategies is computationally cheap and introduces little variance. However, it treats each column independently, ignoring possible relationships between features, and can create artificial spikes in the data distribution that may mislead linear models.

IterativeImputer

This approach fits a regressor (default BayesianRidge) to predict each feature with missing values from the other features. It works well when features are strongly correlated. Because it is an iterative process, you must set random_state for reproducibility. The imputer is sensitive to feature scaling and collinearity; scaling numeric data before this step is often advisable.

Native Handling in HistGradientBoosting

HistGradientBoostingClassifier and HistGradientBoostingRegressor can split on missing values directly, learning the optimal direction for np.nan during tree construction. This removes an entire preprocessing step and the associated leakage risk. The caveat is that the input array must be of float dtype; integer columns cannot contain np.nan, and the technique is unavailable for estimators outside the HistGradientBoosting family.

Implementation Example: Mixed‑Type Pipeline

To combine numeric and categorical columns while keeping the imputation inside the cross‑validation loop, wrap each column type in a Pipeline and combine them with a ColumnTransformer.

# Run in a Python interpreter with scikit‑learn installed
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

# Example column names
numeric_features = ['age', 'income']
categorical_features = ['city', 'department']

# Numeric branch: impute then scale
numeric_pipe = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

# Categorical branch: impute then one‑hot encode
categorical_pipe = Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])

# Assemble the preprocessor
preprocessor = ColumnTransformer([
    ('num', numeric_pipe, numeric_features),
    ('cat', categorical_pipe, categorical_features)
], remainder='drop')

# Final pipeline with estimator
clf = Pipeline([
    ('prep', preprocessor),
    ('est', LogisticRegression(max_iter=1000))
])

# Fitting the pipeline ensures that imputer statistics are derived
# only from the training fold during cross‑validation.

Validation and Limitations

To verify that a SimpleImputer step has removed NaNs, transform a small sample and check:

import numpy as np
X_transformed = preprocessor.fit_transform(X_sample)
has_nan = np.isnan(X_transformed).any()
print('Any NaNs remaining:', has_nan)  # Should be False

For native handling, confirm that HistGradientBoostingClassifier accepts np.nan:
from sklearn.ensemble import HistGradientBoostingClassifier
model = HistGradientBoostingClassifier(random_state=0)
# X_train must be float dtype; np.nan values are allowed
model.fit(X_train, y_train)  # Should not raise ValueError

Limitations to keep in mind

IterativeImputer’s default max_iter and tol can change between scikit‑learn releases, affecting convergence; always inspect the estimator’s attributes or fix them explicitly.
Native missing‑value support in HistGradientBoosting does not treat integer sentinel values (e.g., -999) as missing; convert such sentinels to np.nan before fitting.
Fitting an imputer on the full dataset before splitting the data leaks information and leads to overly optimistic performance estimates; always place the imputer inside a Pipeline or ColumnTransformer that is itself inside cross‑validation.