Kaggle Notebook Accelerator Change and Package Persistence

Does changing the notebook accelerator persist custom pip installations across kernel restarts?

0 reputation · 27 Oct 2023, 21:52 UTC

108.5K views0

Kaggle Notebooks allow users to switch between CPU, GPU, and TPU accelerators via the settings pane. This transition triggers a full kernel restart to reallocate the underlying hardware resources.

While files written to the /kaggle/working directory are preserved during a kernel restart, the state of the Python environment is volatile. Specifically, there is uncertainty regarding whether packages installed via pip during a session remain available after the environment switches accelerators, or if the container is fully reset to the base image.

What is the expected behavior for runtime-installed libraries when transitioning from a CPU to a GPU environment? Does the session maintain the installed package state, or must the installation process be repeated after the accelerator change?

4,886 following

Keep exploring this topic

Kaggle CLI submission file size validation

The Kaggle CLI lacks client-side file size validation for competition submissions, relying on server-side HTTP 400 errors after the upload is complete.

After Effects GPU‑Accelerated Renderer and Adobe Media Encoder Dynamic Link: Memory Leak Persistence?

After Effects GPU‑accelerated renderer fix may not prevent memory leaks when compositions are sent to Media Encoder via Dynamic Link, raising questions about automatic CPU fallback.

Which directory permissions apply to /kaggle/working during notebook commits?

An inquiry into the disk quota and write permissions of the /kaggle/working directory during the notebook commit process compared to interactive sessions.

Does DataSpell’s Local History preserve Jupyter kernel state after notebook restoration?

Exploring whether DataSpell’s Local History can restore Jupyter notebooks to a runnable state, including kernel reconnection, snapshot retention, and programmatic revert options.

Why packages vanish when you change the accelerator

In Kaggle’s runtime, switching from CPU to GPU or TPU triggers a full container teardown. The Python environment is rebuilt from the base image, so any site‑packages added during the session are discarded. Only the /kaggle/working folder is preserved across the restart.

To keep a library available after the switch, re‑run the pip install command at the start of each notebook or place it in a startup.sh script that Kaggle executes automatically. This guarantees the package is installed in the fresh container before any code runs.

Example:

# startup.sh
pip install --quiet my‑custom‑lib

Does changing the notebook accelerator persist custom pip installations across kernel restarts?

0 answers

Your perspective belongs here.

1 question comment

Why packages vanish when you change the accelerator