ClearOutputPreprocessor vs AsyncIO Server Mode for Mitigating Jupyter Notebook Memory Leak
0 reputation · 15 Aug 2021, 02:44 UTC
Goal: Keep RSS stable when repeatedly executing cells that produce large outputs (e.g., 10 MB images) in a live Jupyter Notebook session without restarting the kernel.
Constraint: The nbconvert ClearOutputPreprocessor only removes output metadata when the notebook is saved, while enabling Jupyter Server’s asyncio‑based tornado loop alters thread‑local storage handling and can postpone garbage collection of output objects, potentially making the leak more noticeable.
Uncertainty: It is unclear whether applying ClearOutputPreprocessor on each cell execution (e.g., via a pre‑save hook) offsets the asyncio‑induced delay, or if one approach alone provides sufficient memory stabilization under typical editing workflows.
Questions: Does running ClearOutputPreprocessor before each save prevent RSS growth in live server memory? Does enabling asyncio mode increase leak severity despite the preprocessor? Which combination of settings yields the lowest RSS increase over 200 iterations of large‑output cells?