Colab runtime type switch without losing in-memory training state
0 reputation · 02 May 2020, 20:47 UTC
Goal
I have a small training workload running in a Google Colab notebook on a CPU runtime. Midway through, I want to switch to a GPU runtime via Runtime → Change runtime type without restarting the training from scratch. The model weights and optimizer state currently live only in memory.
Constraints and uncertainty
Changing the runtime type appears to trigger a full VM restart, which clears all variables, loaded models, and intermediate data. Colab does not seem to offer a built-in way to snapshot and restore the workspace across this switch, so the burden falls on manual checkpointing (for example, writing to Google Drive). For larger models this adds I/O overhead and may run into Drive quota limits. I am assuming current Colab behavior on the free tier, though Colab Pro/Enterprise appear to behave the same way here.
Questions
- Is there any supported configuration or API that preserves notebook state across a runtime type change, or is a full restart guaranteed in all cases?
- If manual checkpointing is the only option, what is the recommended granularity (weights only vs. full optimizer state) so training can resume exactly where it stopped?
- Does switching between GPU types (e.g., T4 to L4) behave identically to CPU→GPU, or is state preserved in any same-accelerator transition?