DataSpell 2023.3 kernel cleanup: should heap-threshold warnings be configurable?

Answer

Yes – the heap‑usage threshold that triggers a warning (and optionally an automatic kernel restart) should be made configurable in DataSpell 2023.3.

Confirmed facts

• DataSpell 2023.3 added a background monitor that cleans up orphaned Jupyter kernels only when a notebook is closed.
• The IDE still relies on a static -Xmx value (default 2 GB) and does not intervene during active computation.
• If the heap grows beyond physical RAM, the OS may start swapping, degrading performance more than the original leak.
• External kernels launched outside DataSpell’s built‑in Jupyter integration are not currently monitored by this internal heap tracker.

Likely explanation

Introducing a user‑definable percentage of -Xmx (e.g., 80 %) as a warning threshold would let the IDE detect excessive heap usage before swapping or an OOM kill occurs. A warning‑only approach avoids the risk of losing unsaved state in long‑running analyses, while an optional auto‑restart can be offered for users who accept that trade‑off. External kernels should remain out of scope unless the user explicitly opts‑in, to prevent unintended interference with independently managed environments.

Steps to enable the feature (if implemented)

1. Open Settings → Build, Execution, Deployment → Debugger → DataSpell → Jupyter.
2. Locate the Kernel heap‑usage threshold setting.
3. Enter a percentage (default suggestion: 80) or leave blank to disable.
4. Choose Warn only or Warn and auto‑restart (the latter not recommended for critical workloads).
5. Apply and restart the IDE if required.

Missing diagnostic detail

To confirm that the warning will be accurate, we need to know whether DataSpell can obtain real‑time heap usage of the kernel JVM (e.g., via JMX or OS process metrics). If this measurement is unreliable, the recommendation might shift to disabling the feature until a robust monitoring method is available.