Choosing Between Ephemeral Disk and Google Drive in Colab Notebooks

Decision and Constraints

When you run a notebook in Google Colab you must decide where to store code, intermediate data, and final results. The decision is driven by three practical constraints:

• Free‑tier runtimes disconnect after ~90 minutes of idle time and the virtual machine is recycled.
• The runtime’s local disk (/content) is cleared on every restart, giving you a clean slate but losing any data left there.
• Google Drive provides persistence across sessions, but accessing it incurs authentication overhead, higher latency, and API rate limits.

The goal is to keep fast local throughput for compute‑intensive steps while preserving reproducibility and sharing ability for datasets, models, and notebooks.

Options Comparison

Feature	Ephemeral Runtime Disk (/content)	Google Drive Mount (/content/drive/MyDrive)
Latency	Low (local SSD‑like)	Higher (network‑bound, ~10‑100 ms per op)
Persistence	Lost on runtime restart or disconnect	Persists across sessions and is visible from Drive UI
Auth required	None	OAuth token for the notebook’s session
Typical use	Intermediate artifacts, large temporary files, training checkpoints	Datasets, model exports, notebooks, shared resources
Rate limits	None	Drive API quotas (≈1 GB/day upload for free accounts, per‑second request limits)

Trade‑offs

Using only the runtime disk maximizes I/O throughput for tight loops (e.g., writing batches during model training) but forces you to manually copy anything you need to keep before the runtime expires. Relying solely on Drive guarantees continuity and easy sharing, yet frequent small‑file reads/writes can become a bottleneck and may trigger quota errors. A hybrid approach—keeping hot data in /content and periodically syncing checkpoints or final outputs to Drive—captures the strengths of both while mitigating their weaknesses.

Concrete Implementation Pattern

The following snippet shows a typical workflow: mount Drive once per session, keep working files in /content, and copy results to a Drive folder for persistence.

# Run once per notebook session (requires user auth)
from google.colab import drive
drive.mount('/content/drive')

# Example: copy a dataset from Drive to the fast runtime disk
!cp -r '/content/drive/MyDrive/datasets/imagenet_subset' '/content/data'

# Perform compute‑intensive work under /content
import os, time, numpy as np
# Simulate a large temporary array
big_array = np.random.rand(4000, 4000).astype('float32')  # ~64 MB
np.save('/content/temp_features.npy', big_array)

# Persist the result to Drive
!cp '/content/temp_features.npy' '/content/drive/MyDrive/results/temp_features.npy'

# Verify that the file exists in Drive
print('Drive file list:', os.listdir('/content/drive/MyDrive/results'))

After execution, you can restart the runtime (Runtime → Restart runtime) and confirm that /content is empty while the file remains in Drive.

Verification Steps

1. After mounting, list the Drive root to ensure the mount succeeded:

   import os
   print(os.listdir('/content/drive/MyDrive'))

2. Check free space on the runtime disk and compare write speed (optional, for debugging):

   import shutil, time
   usage = shutil.disk_usage('/content')
   print('Free:', usage.free // (2**30), 'GB')
   # Timed write to local disk
   start = time.time()
   with open('/content/speed_test.bin', 'wb') as f:
       f.write(b'0' * 100_000_000)  # 100 MB
   print('Local write time:', time.time() - start, 's')
   # Timed write to Drive (same size)
   start = time.time()
   with open('/content/drive/MyDrive/speed_test.bin', 'wb') as f:
       f.write(b'0' * 100_000_000)
   print('Drive write time:', time.time() - start, 's')
   

3. Restart the runtime and verify persistence:

   # After restart, run this in a new code cell
   print('Content dir:', os.listdir('/content'))  # should be empty or only Colab defaults
   print('Drive results:', os.listdir('/content/drive/MyDrive/results'))  # should show your file
   

Limitations and Practical Checks

• Idle timeout and disk size can change with Colab updates; treat the ~90 minute figure as a guideline, not a guarantee.
• The mount token lives for the notebook session; avoid committing the notebook with the auth code or hard‑coded paths that could expose credentials.
• Frequent small‑file operations on Drive may hit rate limits; batch writes (e.g., tar or zip) and compress large artifacts when possible.
• Always verify that expected files appear in Drive after a copy step and that they survive a runtime restart.