Shape mismatch in Pipeline with ColumnTransformer

Ensuring Constant Feature Dimensionality

To ensure the output feature count remains constant across different data splits, configure the OneHotEncoder within your ColumnTransformer with handle_unknown='ignore'. This ensures that the number of columns generated during the .transform() phase exactly matches the number of columns established during the .fit() phase, regardless of whether the test set contains new, unseen categories.

The Role of handle_unknown

In a scikit-learn Pipeline, the estimator (e.g., RandomForestClassifier or LinearRegression) expects a fixed input shape based on the training data. The handle_unknown parameter in OneHotEncoder dictates how the transformer reacts to categories in the test set that were not present during training:

• 'error' (Default): The pipeline raises a ValueError when an unknown category is encountered, halting the prediction process.
• 'ignore': The encoder represents the unknown category as a row of all zeros across the one-hot encoded columns for that feature. It does not create new columns for the unknown categories.

Because 'ignore' prevents the creation of new columns and avoids crashing, the downstream estimator receives a feature matrix with the exact dimensionality it was trained on, eliminating shape mismatch errors.

Implementation Pattern

Assuming scikit-learn 1.0+, the following configuration is the standard approach for maintaining alignment:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier

# Define the transformer with handle_unknown='ignore'
preprocessor = ColumnTransformer(
    transformers=[
        ('cat', OneHotEncoder(handle_unknown='ignore'), ['categorical_column'])
    ]
)

# Encapsulate in a pipeline
clf = Pipeline(
    steps=[
        ('preprocessor', preprocessor),
        ('classifier', RandomForestClassifier())
    ]
)

Verification Steps

To verify that your pipeline is resilient to shape mismatches, perform the following check:

1. Fit the pipeline on a training set containing a specific set of categories (e.g., ['A', 'B']).
2. Pass a test sample containing a completely new category (e.g., ['C']) to the .predict() method.
3. Confirm that the prediction completes without a ValueError and that the internal transformation output maintains the same column count as the training set.

Diagnostic Note: If you are still experiencing shape mismatches despite using handle_unknown='ignore', please specify if you are using any custom transformers or FunctionTransformer steps that may be dynamically altering the array shape based on input values.