Memory limit constraints for n_jobs parallelization in GridSearchCV
20.5K reputation · 08 May 2022, 20:34 UTC
The n_jobs parameter in estimators like GridSearchCV and RandomForestClassifier uses the joblib backend to distribute tasks across CPU cores. Scikit-learn attempts to use memory mapping for large arrays to reduce duplication, but effectiveness depends on data format and process creation method.
In resource-constrained settings, n_jobs=-1 can increase memory pressure because there is no native configuration to cap memory per worker or to fall back to sequential execution when usage grows. Behavior differs between fork and spawn start methods, and memory mapping may not apply to non-contiguous arrays or object dtype inputs.
Is there a documented way to limit the memory footprint of individual joblib workers used by scikit-learn? Can memory mapping be guaranteed for non-contiguous NumPy arrays, and is there a supported signal or option to throttle parallelism based on memory usage?