Duplicate or missing entries when paginating Airflow REST API with limit/offset

Answer

No. Airflow’s stable REST API v2.0+ does not guarantee stable, exactly-once ordering for limit/offset pagination when dag runs are created concurrently with paging. Duplicate or skipped entries can be observed.

Confirmed facts: The stable API uses offset/limit pagination with a server default limit of 100 and a configurable maximum, typically 1000 via api.max_page_limit. Ordering is controlled by order_by, commonly execution_date descending, but the API does not provide snapshot isolation or a cursor token. The UI and CLI apply their own bounding limits separate from the API.

Likely explanation for duplicates / misses

Under concurrent writes the result set shifts between requests. With offset-based paging:

• If new dag runs are inserted with a sort key that places them before the current page, the offset for the next request points past items that moved down, causing duplicates.
• If new dag runs are inserted with a sort key that places them after the current page but before the next offset window, items can be skipped.

This is a property of offset pagination without a stable snapshot, not a bug in Airflow. Scheduler activity, dynamic DAGs, timetable triggers and backfills can create dag runs while a client is paging.

Client-side mitigations without cursor pagination

1. Page over a fixed date window. Use start_date_gte / start_date_lte or end_date_gte / end_date_lte with UTC timestamps that match Airflow storage. Keep the window immutable for the duration of paging.
2. Fix ordering explicitly. Pass a deterministic order_by, e.g. -execution_date or execution_date, and avoid ordering by non-unique fields alone.
3. Deduplicate by dag_run_id in the client. Collect seen IDs across pages and drop repeats; track the minimum/maximum execution_date seen to detect gaps.
4. Reduce page size and latency. Smaller limit values reduce the time window for intervening inserts.

Assumptions: Airflow 2.0+ stable REST API, endpoint such as GET /api/v2/dags/{dag_id}/dagRuns with limit/offset parameters. Behavior varies by version: defaults and max_page_limit changed across 2.0-2.4 and were standardized in 2.5+ OpenAPI spec. Database transaction isolation is READ COMMITTED by default, so newly committed runs become visible mid-page.

One diagnostic detail that changes the recommendation: which Airflow version and which list endpoint you are paging, and whether api.max_page_limit has been changed from the default. Confirm with `airflow version` and GET /api/v2/config for api.max_page_limit.