Automate Kaggle Dataset Downloads and Competition Submissions

Desired outcome

You need a Kaggle dataset inside an automated script or CI pipeline, or you want a build job to submit competition predictions without opening a browser. Manual downloads through the Kaggle website do not scale to repeated or scheduled runs.

The useful takeaway is to install the Kaggle Python API package, authenticate once with your Kaggle username and API key, then use the kaggle command-line tool to download datasets and submit competition files. This guide covers credential setup, dataset download, submission preparation, and verification.

Prerequisites

• A Kaggle account with a verified email address.
• Python 3 and pip installed where the script will run.
• Network access to kaggle.com from the execution environment.
• Write permission to the directory where datasets will be extracted.

Install and configure authentication

The Kaggle API authenticates with a username and API key, not your account password. Generate the key on the Kaggle account settings page under the API section. The browser downloads a file named kaggle.json.

Install the current release of the kaggle package:

pip install kaggle

On Linux or macOS, run the following in a local terminal to place the credential where the CLI expects it:

mkdir -p ~/.kaggle
cp ~/Downloads/kaggle.json ~/.kaggle/kaggle.json
chmod 600 ~/.kaggle/kaggle.json

The chmod 600 command restricts read and write access to the current user. Treat this file as a long-lived secret: do not commit it to version control or paste it into chat or logs.

In CI/CD environments, set KAGGLE_USERNAME and KAGGLE_KEY as environment variables instead of writing a credential file. The API package reads these variables automatically.

Verify authentication by listing available datasets:

kaggle datasets list

A successful run prints dataset entries and exits with code 0. An authentication error means the credential file is missing, malformed, or unreadable.

Download a public dataset

Use the dataset owner and slug found in the dataset URL. For example, if the URL is kaggle.com/datasets/owner/dataset-name, the owner is owner and the dataset identifier is owner/dataset-name.

Run from the project directory where you want the data:

kaggle datasets download -d <owner>/<dataset> --unzip --path ./data

--unzip extracts archives after download. --path ./data writes files to a local data directory. Without --path, files extract into the current working directory.

Expected check: the data directory contains the extracted files, and the command exits with code 0. Relevant risks: large archives can exhaust disk space on constrained build agents, and re-running the command may overwrite files already in the target directory.

Prepare a competition submission file

Kaggle validates submission files against the competition's sample submission schema. Download the sample submission from the competition's Data tab and match its column names, order, and row count exactly. Do not add extra columns or reorder rows.

Save your predictions to a file such as submission.csv in the project root.

Submit to a competition

Find the competition slug in its URL, for example kaggle.com/competitions/competition-name makes the slug competition-name. Run from the directory containing the submission file:

kaggle competitions submit -c <competition> -f submission.csv -m "Automated submission from CI"

Required permissions: a Kaggle account that has accepted the competition rules. The -m flag adds a human-readable message to the submission history.

Expected check: open the competition's Submissions page on Kaggle and confirm the entry appears with the message you provided. A non-zero CLI exit or a rejection notice usually means a schema mismatch, missing headers, or incorrect row count.

Expected checks and recovery

Step	Verification	Likely failure
Authentication	kaggle datasets list returns entries	Missing or unreadable ~/.kaggle/kaggle.json
Dataset download	Extracted files exist in --path directory	Insufficient disk space or network timeout
Competition submission	Submission appears on Kaggle Submissions page	Schema mismatch or competition rules not accepted

If a credential is exposed, regenerate it on the account settings page and replace the old file immediately. If a submission is rejected, compare your file header against the sample submission downloaded from the competition page before retrying.

Limitations

• Kaggle enforces rate limits and terms of service; excessive automated access may be blocked.
• Private datasets require manual license acceptance on the website before the API can download them.
• API keys are long-lived secrets and must be protected like passwords.
• Kaggle may change API behavior, authentication, or rate limits without notice.

There is no rollback for a submitted competition entry. If you upload the wrong file, submit a corrected file before the competition deadline; the leaderboard typically uses your best valid submission.