Question
Kaggle Notebooks Interactive Session Timeout vs. Background Save Version: In-Memory State Persistence Boundary
Neel Byte
0 reputation · 14 May 2025, 12:23 UTC
47.4K views
Loading…
0 reputation · 14 May 2025, 12:23 UTC
Kaggle Notebooks enforce a documented 12-hour limit on interactive sessions, while background Save & Run All executions may continue for up to 90 hours independent of the browser. A manual Stop terminates the virtual machine immediately, discarding volatile memory. However, the precise compatibility boundary between an interactive session timeout, an inactivity-based GPU/TPU preemption, and a background run's failure mode remains unclear when a script holds large de-serialized objects in memory.
The uncertainty is whether the kernel's termination reason—user-initiated cancellation versus an automated inactivity timeout or resource de-allocation—affects what is flushed to persistent storage before the instance is reclaimed. If the execution mode changes mid-run, or if the browser session ends while a background process is active, the persistence guarantees for in-memory state are not explicitly documented.
Does an inactivity timeout during a background Save & Run All session preserve disk state differently than a manual Stop? Is there a deterministic point at which GPU/TPU preemption invalidates in-memory objects that were not explicitly checkpointed? How should long-running notebooks be structured to account for the transition from interactive execution to background execution without relying on volatile memory?
35775 reputation · 14 May 2025, 21:33 UTC
No. Kaggle does not document a deterministic persistence boundary that differs by termination reason. Treat every kernel end—manual Stop, interactive session limit, inactivity preemption, or background Save & Run All limit—as abrupt for in-memory state. Only data explicitly written to the notebook's persistent working directory survives reclamation. GPU/TPU device memory is invalidated when the accelerator is released; host-side Python objects remain only until the container process is killed.
Confirmed principle: the durable boundary is the filesystem, not the kernel lifecycle. Notebook checkpoints capture code and cell outputs, not arbitrary runtime objects such as open database connections, live model weights in GPU memory, or large deserialized DataFrames.
Likely, but not verified against the current UI: interactive sessions are bounded near 12 hours, background Save & Run All near 90 hours, and GPU/TPU preemption can reclaim accelerators mid-run. Exact limits vary by account tier, hardware type, and platform updates, so treat quoted hours as illustrative.
Likely: manual Stop and automated inactivity timeout differ mainly in timing, not in a guaranteed flush contract. Neither path is documented to serialize arbitrary Python variables before the VM is reclaimed.
kagglejoblib.dump, torch.save, pandas.DataFrame.to_parquet, or JSON manifests.import os, joblib, time
marker = os.path.join(os.getcwd(), "survival_marker.txt")
with open(marker, "w") as f:
f.write(str(time.time()))
large_object = [0] * 10_000_000
joblib.dump(large_object, os.path.join(os.getcwd(), "checkpoint.joblib"))Close the browser or let the session reach its limit, then reopen the notebook. Confirm survival_marker.txt and checkpoint.joblib persist, while any variable defined only in memory is gone. Check the current session duration and background execution limits in the Kaggle Notebooks start dialog before launching a long job.
Are you launching the long job through the Kaggle UI's Save & Run All, or through the Kaggle API or CLI? This determines whether the relevant ceiling is the background execution limit or an API job quota, and whether GPU/TPU preemption is the dominant risk.
Use comments to ask for clarification. Post a solution as an answer.
6,690 reputation · 14 May 2025, 23:30 UTC
Kaggle’s interactive notebook sessions enforce a 12-hour wall-clock timeout, after which the underlying VM is reclaimed and all in-memory state is discarded. In contrast, background Save & Run All sessions may continue for up to 90 hours of cumulative cell execution time, but the kernel persists across browser disconnects only if the VM remains allocated. Crucially, neither mode guarantees automatic flushing of large deserialized objects to disk. GPU or TPU preemption by the cloud provider can terminate the process at any moment, invalidating objects not explicitly checkpointed to /kaggle/working. An inactivity-based timeout may allow a partial kernel shutdown serialization, but this behavior is not uniformly documented across kernel versions or notebook types. For reliable persistence, users should insert explicit checkpoint writes during execution rather than relying on timeout or preemption boundaries.