Kernel termination during Kaggle Notebook timeout prevents state persistence

Answer to the Core Question

To trigger a checkpoint reliably before a Kaggle kernel is abruptly killed, you need a two‑step strategy:

1. Estimate remaining time. Kaggle imposes a hard limit of 12 h on CPU kernels and 9 h on GPU kernels. In the absence of an official API, record the start time with time.time() and subtract it from the known limit, leaving a safety buffer (e.g., 5 min). If the remaining time falls below the buffer, call your checkpoint routine immediately.
2. Write atomically and flush. Use a temporary file, flush the buffer, sync the file descriptor, and then rename it to the final checkpoint path. This guarantees that if the kernel dies mid‑write, the old file remains untouched and the new file is either fully present or absent.

Concrete Implementation

Below is a minimal, reproducible pattern you can drop into a training loop:

# -*- coding: utf-8 -*-
import time, os, tempfile, shutil
import torch

# 1. Time estimation
KERNEL_LIMIT = 12 * 60 * 60   # 12 hours in seconds (CPU)
BUFFER = 5 * 60              # 5 minute safety buffer
START = time.time()

# 2. Checkpoint helper

def checkpoint(model, epoch, path):
    """Save  atomically to .
    The function writes to a temp file, flushes, fsyncs, and renames.
    """
    dirpath = os.path.dirname(path)
    os.makedirs(dirpath, exist_ok=True)
    with tempfile.NamedTemporaryFile(dir=dirpath, delete=False) as tmp:
        torch.save({'epoch': epoch, 'model_state': model.state_dict()}, tmp.name)
        tmp.flush()
        os.fsync(tmp.fileno())   # ensure data is on disk
    os.replace(tmp.name, path)   # atomic rename

# 3. Training loop skeleton
for epoch in range(num_epochs):
    # ... training code ...

    # Periodically check time
    elapsed = time.time() - START
    remaining = KERNEL_LIMIT - elapsed
    if remaining <= BUFFER:
        checkpoint(model, epoch, f'/kaggle/working/checkpoint_epoch{epoch}.pt')
        break   # optional: stop training to avoid abrupt kill

Why This Works

• Time estimation uses only the known limit; no hidden kernel signal is required.
• Atomic rename guarantees that a partially written file never replaces a valid checkpoint.
• Flush + fsync forces the OS to commit the data to the underlying storage before the rename.
• All operations happen inside /kaggle/working, which survives kernel restarts.

Separating Fact from Likely Cause

Confirmed facts:

• Kernel termination is immediate and does not run finally blocks.
• Only files in /kaggle/working persist across restarts.
• Unflushed writes can be truncated if the process dies.

Likely cause (not guaranteed):

• Some Kaggle environments expose a KAGGLE_KERNEL_TIMEOUT variable; if present, you can use it instead of hard‑coding the limit.

Ask for One Diagnostic Detail

To fine‑tune the safety buffer, could you confirm whether you are running a CPU or GPU instance? The exact timeout differs (12 h vs 9 h) and will affect the KERNEL_LIMIT value above.