Download Kaggle Datasets in CI Without Committing Your API Token

Manual downloads break automated training

You need a public Kaggle dataset inside an automated training pipeline, but Kaggle’s download button opens a browser. Downloading the file by hand makes the pipeline non-reproducible, and committing kaggle.json to version control exposes a long-lived credential to anyone with repository access. The practical alternative is to authenticate the Kaggle CLI with environment variables and validate the connection before downloading.

This post covers how to configure the Kaggle API token without writing it to disk, run a download in a CI runner or local terminal, and verify that authentication succeeded. It assumes you are using the official Kaggle Python package and CLI.

How Kaggle API authentication works

Kaggle issues a user-specific API token as a JSON file named kaggle.json. It contains your Kaggle username and a long-lived API key. The CLI and Python package look for this file at ~/.kaggle/kaggle.json, but they also accept the same credentials through environment variables:

Method	Where credentials live	Best suited for
Token file	~/.kaggle/kaggle.json	Local workstation or notebook
Environment variables	KAGGLE_USERNAME and KAGGLE_KEY	CI/CD pipelines and ephemeral runners

Treat the token like a password. Never echo it in build logs, and never add kaggle.json to Git. If you use the file method, restrict permissions so only your user can read it.

Worked example: download a dataset in CI

Run these commands on a CI runner or local terminal with network access to PyPI and Kaggle. Replace placeholders in angle brackets with your values.

1. Install the Kaggle package:

pip install kaggle

1. Provide credentials without writing them to disk:

export KAGGLE_USERNAME=""
export KAGGLE_KEY=""

Required permissions: the runner must be able to write to the target download directory and read the environment variables. If you instead place kaggle.json at ~/.kaggle/kaggle.json, set its permissions to 600.

1. Verify authentication before downloading:

kaggle datasets list --search "" --limit 1

Expected check: the command exits with code 0 and prints at least one dataset row. A 401 or 403 response means the token is missing, invalid, or lacks access to the requested resource.

1. Download and unzip the dataset:

kaggle datasets download -d / -p ./data --unzip

Expected check: files appear in ./data, such as CSV or Parquet files matching the dataset metadata. Risks include network timeouts for large files, HTTP 429 rate-limit responses, and silent failures if the runner lacks internet access.

Limitations and failure diagnosis

• Private datasets: You must accept the dataset terms on the Kaggle website before the API can download it. Until then, the CLI returns an access error even with a valid token.
• Kaggle Notebooks: Outbound internet access depends on the notebook’s internet setting and kernel type. An offline kernel cannot run these commands.
• Rate limits: Kaggle does not publish fixed API rate limits, and they can change. Automated jobs should catch HTTP 429 and retry with backoff rather than failing immediately.
• Competition submissions: If you later extend this pattern to competition submissions, per-day limits vary by competition and require an accepted competition entry.

Practical way to confirm the setup works end-to-end: run kaggle datasets list --search "titanic" on the target machine. If it returns results without a browser login, authentication and network reachability are working.

Actionable closing

1. Generate or rotate your API token from your Kaggle account settings under “API.”
2. Store KAGGLE_USERNAME and KAGGLE_KEY as secrets in your CI provider, not in the repository.
3. Add a lightweight authentication step, such as kaggle datasets list --search <keyword>, before any download step in the pipeline.
4. Rotate the token immediately if it is accidentally printed to logs or committed.

These steps reflect the current Kaggle CLI behavior as described in public documentation. Because Kaggle can change authentication or rate-limit behavior without notice, verify the token format and error handling in your own account settings before relying on this in production.