Persisting Data in Google Colab: Mounting Google Drive for Reliable Storage

Why Mount Google Drive?

Google Colab runtimes are temporary virtual machines. When a session ends, any files written to /content disappear. For long‑running experiments, model checkpoints, and shared datasets you need a persistent store. Google Drive offers a convenient, cloud‑based file system that can be accessed directly from a Colab notebook via the google.colab library.

How to Mount and Use Drive in Colab

1. Import the library:

   from google.colab import drive
   

2. Mount the drive to the standard path /content/drive.

   drive.mount('/content/drive')
   

   When this command runs, Colab opens an OAuth2 consent screen. Grant access, copy the provided code, and paste it back into the notebook cell. After successful authentication, a symbolic link is created that points /content/drive to the root of your Google Drive.

3. Verify the mount:

   !ls /content/drive/MyDrive
   

   The output should list the contents of your Drive’s MyDrive folder.

Practical Example: Saving a Model Checkpoint

Below is a minimal PyTorch training loop that writes a checkpoint to the mounted drive. The checkpoint survives a runtime restart, demonstrating persistence.

import torch
import torch.nn as nn
import torch.optim as optim

# Simple network
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(10, 1)
    def forward(self, x):
        return self.fc(x)

model = Net()
optimizer = optim.SGD(model.parameters(), lr=0.01)
criterion = nn.MSELoss()

# Dummy data
x = torch.randn(4, 10)
y = torch.randn(4, 1)

for epoch in range(5):
    optimizer.zero_grad()
    out = model(x)
    loss = criterion(out, y)
    loss.backward()
    optimizer.step()
    # Save checkpoint
    checkpoint_path = '/content/drive/MyDrive/checkpoints/epoch_{}.pt'.format(epoch)
    torch.save(model.state_dict(), checkpoint_path)
    print(f"Saved epoch {epoch} to {checkpoint_path}")

After the loop, you can inspect MyDrive/checkpoints in the Drive web UI or via another notebook cell to confirm the files exist. Restart the runtime, remount the drive, and reload the state dict to resume training.

Limitations and Common Pitfalls

• Network Latency: Drive I/O is performed over the network. Frequent small‑file reads/writes can be noticeably slower than local /content storage. Batch your I/O or move large data files to /content before training.
• API Rate Limits: Google Drive imposes per‑second request limits. When looping over thousands of files (e.g., for dataset shuffling) you may hit 429 Too Many Requests. Use time.sleep() or cache files locally.
• Authentication Expiry: The OAuth2 token is tied to the Colab session. If the runtime restarts, you must re‑run drive.mount() and re‑enter the code. Automate this step in notebooks that are often re‑loaded.
• Permission Issues: Ensure the Drive folder you target is owned by the same Google account that authorized the notebook. Shared folders require explicit permission settings.
• File System Quotas: Google Drive has a 15 GB free quota (plus any Google Workspace allotment). Exceeding this can cause writes to fail.

Best Practices for Large Datasets

• Store raw data in Drive, but copy it to /content once per session using shutil.copytree() or rsync to benefit from faster local I/O during training.
• Compress large files (e.g., .zip or .tar.gz) before uploading to reduce API calls.
• Use google.colab.files.upload() for small auxiliary files that don’t merit Drive storage.
• Clean up /content/drive after training to avoid cluttering your Drive.

Verification Checklist

1. Run drive.mount('/content/drive') and confirm the OAuth flow completes.
2. Execute !ls /content/drive/MyDrive to see your Drive contents.
3. Write a test file: !echo 'test' > /content/drive/MyDrive/test.txt and verify it appears in the Drive UI.
4. Restart the runtime, remount, and confirm test.txt still exists.

Following these steps guarantees that your experiment data remains intact across Colab sessions.