Diagnosing Notebook Reproducibility Issues on Kaggle: When Save & Run All Fails

Recognizable Condition

In a Kaggle notebook you may see that running all cells interactively (Run All) completes in a few minutes and produces clean output. However, when you click Save & Run All to commit the notebook, the run times out, crashes, or returns different predictions. This discrepancy is a common reproducibility problem that can stem from stateful dependencies, resource limits, nondeterminism, or environment drift.

Root Causes & Quick Summary Table

Cause	Why It Happens
Stateful Dependencies	Interactive sessions keep variables, cached downloads, or GPU memory across cells; a fresh commit run starts from scratch.
Resource Limits	Commit runs expose peak memory or GPU usage that interactive incremental runs hide.
Nondeterminism	Unseeded random generators or GPU stochastic ops produce varying outputs.
Environment Drift	Packages installed in the session are not pinned, so the clean commit environment may lack them or use newer versions.

Ordered Diagnostic Checks

1. Confirm the anomaly is commit‑only. Run the notebook interactively, then run Save & Run All on a copy. Compare the logs; if the interactive run succeeds but the commit fails, proceed to the next step.
2. Check for stateful dependencies.
   • Search for global variables or cached objects that are not re‑initialized in each cell.
   • Verify that all data files are downloaded or loaded within the notebook (e.g., !wget or !git clone inside a setup cell).
   • Use !ls -l to confirm file presence before any processing step.
3. Verify resource usage.
   • In an interactive cell, run !nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits to capture GPU memory.
   • For RAM, use !ps -o rss -p $$ | tail -n +2 or !python -c "import psutil, os; print(psutil.Process(os.getpid()).memory_info().rss / (1024**2))".
   • Compare these values to the peak memory reported in the commit run’s logs. If the commit run exceeds the quota (usually 12 GB RAM or 4 GB GPU), you must reduce peak usage.
4. Check for nondeterminism.
   • Set explicit seeds before any random operation:

     import random, numpy as np, torch
     random.seed(42)
     np.random.seed(42)
     torch.manual_seed(42)
     if torch.cuda.is_available():
         torch.cuda.manual_seed_all(42)
         torch.backends.cudnn.deterministic = True
         torch.backends.cudnn.benchmark = False
     

   • Re‑run the commit after adding these lines. If outputs now match the interactive run, randomness was the culprit.
5. Compare environments.
   • In an interactive cell, run !pip freeze and !python -c "import sys, torch, numpy; print(sys.version); print(torch.__version__); print(numpy.__version__)".
   • Store the output in a file (!pip freeze > requirements.txt) and add a setup cell that installs these exact versions:

     # Setup cell – run once in the notebook
     !pip install -r requirements.txt
     

   • Run Save & Run All again. If the commit now succeeds, environment drift was the cause.

Fixes Tied to Findings

• Remove stateful dependencies: Ensure every cell is self‑contained: re‑load data, re‑initialize models, and delete unused objects with del and torch.cuda.empty_cache().
• Reduce peak memory: Use generators for data loading, batch the dataset, or switch to mixed‑precision training with torch.cuda.amp.autocast.
• Enforce determinism: Add the seed block shown above at the very top of the notebook. Avoid functions that ignore the seed (e.g., some data augmentation libraries).
• Pin the environment: Commit a requirements.txt or use a custom Docker image (if Kaggle’s private kernels support it) to lock package versions.

Escalation Criteria

If, after performing all the checks and applying the corresponding fixes, the notebook still fails or produces inconsistent results, the problem is likely platform‑side:

• Capture the full error output from the commit run’s logs.
• Record the commit ID and the timestamp of the failing run.
• Open a support ticket with Kaggle, attaching the notebook, the logs, and the steps you have already taken.

Concrete Example

Below is a minimal notebook that demonstrates the problem and its resolution. The notebook trains a tiny CNN on MNIST using GPU. The interactive run works, but the commit run fails because the dataset is not re‑downloaded and the GPU memory spikes.

# Cell 1 – Setup
import os, random, numpy as np, torch
from torch import nn, optim
from torchvision import datasets, transforms

# Seed for reproducibility
random.seed(42)
np.random.seed(42)
torch.manual_seed(42)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(42)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

# Pin environment
!pip install torch==2.1.0 torchvision==0.16.0

# Download MNIST only if not present
if not os.path.exists('mnist_data'):
    os.makedirs('mnist_data', exist_ok=True)
    !wget -q https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz -O mnist_data/mnist.npz

# Load data
mnist = np.load('mnist_data/mnist.npz')
train_images, train_labels = mnist['x_train'], mnist['y_train']

# Preprocess
transform = transforms.Compose([transforms.ToTensor()])
train_dataset = torch.utils.data.TensorDataset(transform(train_images).float(), torch.tensor(train_labels))
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=64, shuffle=True)

# Simple model
class SimpleCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(1, 32, 3, 1),
            nn.ReLU(),
            nn.Flatten(),
            nn.Linear(5408, 10)
        )
    def forward(self, x):
        return self.net(x)

model = SimpleCNN().cuda()
optimizer = optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

# Training loop – single epoch for demo
model.train()
for batch_idx, (data, target) in enumerate(train_loader):
    data, target = data.cuda(), target.cuda()
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    if batch_idx == 0:
        print('Batch 0 loss:', loss.item())
        break

In an interactive session, this runs quickly because the MNIST file is cached and the GPU memory is reused. When you Save & Run All, the notebook starts from a clean state: the mnist_data folder is empty, so the download step executes again, and the GPU memory peaks during the first batch, potentially exceeding the 4 GB limit and causing a timeout. After adding the download guard and the seed block, the commit run succeeds and reproduces the same loss value.

Practical Verification Checklist

• Run !pip freeze in both interactive and commit runs; the outputs should match.
• Verify that the dataset folder exists before loading; if not, the download step should run.
• Confirm that the seed block is the first executable cell; no random calls should precede it.
• Check the commit run’s log for OOM or timeout messages; if present, reduce batch size or clear caches.

Conclusion

Reproducibility on Kaggle hinges on understanding the difference between an interactive session and a committed run. By systematically checking statefulness, resource usage, determinism, and environment consistency, you can pinpoint why a notebook behaves differently and apply the targeted fix. If all local fixes fail, gather logs and reach out to Kaggle support—most platform‑side issues are documented and can be resolved with the right information.