Automating Kaggle Dataset Ingestion with the Data API

Why the Kaggle Data API Matters

When a data‑science team needs fresh data every day, manually downloading from Kaggle’s web UI becomes a bottleneck. The official Kaggle Data API turns the web interface into a programmatic, repeatable source. It handles authentication, pagination, versioning, and metadata so that a CI/CD pipeline can pull the exact dataset version it expects, verify integrity, and feed it into downstream processing.

Key Prerequisites

• Python 3.9+ (the official client is a pure Python package)
• kaggle package – pip install kaggle
• Personal API token generated in the Kaggle web UI and stored in ~/.kaggle/kaggle.json with permissions 600
• Network access to https://www.kaggle.com/api/v1/ (blocked in some corporate firewalls)

Authentication: One Token, No OAuth

The API uses a simple JSON file containing a username and key. The client reads ~/.kaggle/kaggle.json automatically. If you prefer not to store the token in the home directory, set the environment variable KAGGLE_CONFIG_DIR to a secure location:

export KAGGLE_CONFIG_DIR=/secure/dir
# The directory must contain kaggle.json with 600 permissions

Never commit kaggle.json to source control; use a secret manager or CI environment variable to inject it at runtime.

Listing Datasets with Pagination

Large catalog queries can return thousands of items. The datasets/list endpoint supports page and page_size parameters. The client library exposes these via the kaggle.datasets.list() function:

# List the first 20 datasets containing "titanic"
kaggle.datasets.list(search="titanic", page=1, page_size=20)

For scripts that need to iterate over the full catalog, loop until has_more is false:

page = 1
while True:
    result = kaggle.datasets.list(page=page, page_size=50)
    for ds in result:
        print(ds.ref)  # e.g., "zillow/zecon"
    if not result.has_more:
        break
    page += 1

Rate‑limit enforcement is 200 requests per hour. Exceeding this triggers a temporary block, so cache results locally or batch requests.

Downloading a Specific Dataset Version

Datasets may have multiple releases. The client can fetch a particular version with the -v flag. The API returns a ZIP archive that can be extracted or streamed directly into a processing step.

# Download version 1.0 of a dataset
kaggle.datasets.download(dataset_ref="username/dataset-name", version="1.0", path="/tmp/data")
# The ZIP is extracted automatically if you use the --unzip flag
kaggle.datasets.download(dataset_ref="username/dataset-name", version="1.0", unzip=True, path="/tmp/data")

To verify integrity, query the metadata endpoint and compare checksums:

meta = kaggle.datasets.get_info("username/dataset-name", version="1.0")
for file in meta.files:
    print(f"{file.name} – {file.size_bytes} bytes – {file.checksum}")

Integrating Into a Pipeline

Below is a minimal Airflow DAG that pulls a dataset, verifies the checksum, and moves the data to a S3 bucket. The DAG uses the kaggle CLI via a BashOperator for simplicity.

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG("kaggle_ingest", start_date=datetime(2024, 1, 1), schedule_interval="@daily") as dag:
    download = BashOperator(
        task_id="download_dataset",
        bash_command="kaggle datasets download -d username/dataset-name -v 1.0 -p /tmp/data"
    )
    verify = BashOperator(
        task_id="verify_checksum",
        bash_command="python verify.py /tmp/data/dataset.zip"
    )
    upload = BashOperator(
        task_id="upload_to_s3",
        bash_command="aws s3 cp /tmp/data/dataset.zip s3://my-bucket/dataset/"
    )
    download >> verify >> upload

Key points:

• The KAGGLE_CONFIG_DIR variable is set in the Airflow worker environment.
• Validation is performed before uploading to avoid wasting bandwidth.
• All operations are idempotent; the DAG can be retried without side effects.

Trade‑Offs and Limitations

While the API simplifies ingestion, consider the following:

• Rate limits – 200 requests/hour. For high‑volume catalog scans, implement exponential back‑off or cache results.
• Large files – Downloads can be several GB. Use --unzip only if you have sufficient disk; otherwise stream the ZIP into a decompression pipeline.
• Versioning granularity – Some datasets have many minor releases. Decide whether you need the latest or a pinned version.
• Security – The token grants read access to all datasets you own or have been shared with. Rotate tokens regularly and store them in a secret manager.

Actionable Checklist

1. Generate a Kaggle API token and store it in ~/.kaggle/kaggle.json with chmod 600.
2. Install the Python client: pip install kaggle.
3. Test pagination: kaggle datasets list -p 1 -s "titanic".
4. Download a known dataset version and verify the ZIP contents.
5. Embed the download logic into your CI/CD or orchestrator (Airflow, Prefect, etc.).
6. Implement retry logic or caching to stay within the 200‑req/hour limit.

With these steps, you can reliably pull Kaggle data into any automated workflow, ensuring reproducibility and auditability.