Pin Your Kaggle Datasets for Reproducible Notebooks

The Reproducibility Trap on Kaggle

When you're iterating on a machine learning project in Kaggle Notebooks, it's easy to fall into a subtle trap: your dataset changes without you noticing. Someone on your team updates the dataset, or you accidentally overwrite it while experimenting. Your notebook still runs, but now it's training on different data. This breaks reproducibility and can lead to hours of debugging when results don't match expectations.

Versioned Datasets: Immutable Inputs for Notebooks

Kaggle Datasets solve this with versioning. Each time you publish an update via the API, Kaggle creates a new numbered version. The key insight is that notebooks can pin to a specific version, making your experiments truly reproducible. Your notebook references owner/dataset-name:version-3, not just the latest.

When you attach a dataset to a notebook, it mounts read-only under /kaggle/input/dataset-slug. This enforces a clean separation: your code writes outputs to /kaggle/working and never mutates source data. It's like having a version-controlled data lake with built-in immutability.

Automating Dataset Publishing with the API

The real power comes from scripting dataset updates through the Kaggle API. Instead of manually uploading through the web interface, you can integrate dataset publishing into your CI/CD pipeline. This turns data release into an auditable, repeatable step.

The workflow is straightforward: prepare your data, create a dataset-metadata.json file with your dataset's configuration, then run kaggle datasets create -p /path/to/data or kaggle datasets version -p /path/to/data to publish. The slug (owner/dataset-name) remains stable across versions, so downstream notebooks can reference it reliably.

Worked Example: Setting Up Versioned Data Pipeline

Here's a practical example of setting up automated dataset versioning:

1. Create dataset metadata - In your data directory, create dataset-metadata.json:

{{"title": "Training Data v1", "id": "your-username/training-data", "subtitle": "Monthly refreshed dataset", "description": "Processed training data with feature engineering applied", "keywords": ["training", "features", "time-series"], "license-type": "CC0-1.0"}}

1. Initialize dataset via API - Run from your project root:

kaggle datasets create -p ./data/

1. Version updates - After processing new data, update the metadata file with a new subtitle or description, then run:

kaggle datasets version -p ./data/ -m "Updated with October 2026 features"

1. Reference in notebooks - In your notebook, use the versioned dataset URL or specify the version in the dataset selection UI. The data appears at /kaggle/input/your-username-training-data.

Trade-offs and Limitations

While versioned datasets are powerful, they come with constraints. Dataset size limits (currently 20GB total, 5GB per file for most accounts) mean you may need to split large corpora across multiple datasets. File count limits also apply, which can affect projects with many small files.

Be careful with the "latest" reference. If you pin to "latest" in shared notebooks, a teammate's dataset update can silently change your inputs. Always pin to a specific version for production or competition work. The version history on the dataset page shows all available versions and their creation times.

Licensing is another responsibility—the uploader must ensure they have rights to distribute the data. Public datasets are discoverable through Kaggle's search, so good metadata (title, description, column documentation) materially affects whether others can find and reuse your work.

Verify Your Setup

After creating or versioning a dataset, verify it works as expected:

• Check the dataset page UI to confirm the version counter incremented
• Attach the dataset to a notebook and verify files appear under /kaggle/input/
• Confirm writes to /kaggle/input/ fail (read-only enforcement)
• Test referencing a specific version rather than "latest"

For production pipelines, consider adding a verification step to your CI that checks the dataset version matches expectations before running training jobs.