Duplicate or missing entries when paginating Airflow REST API with limit/offset

Question

Determine whether Airflow’s stable REST API limit/offset pagination returns each dag run exactly once when new dag runs are created while a client pages through the collection.

Constraints include the server‑imposed default and maximum page size, the absence of snapshot‑isolated ordering, version‑specific defaults for limit/offset, and the fact that the UI and CLI apply separate bounding limits. Under concurrent scheduling activity, offset‑based paging can produce duplicate or missing entries, forcing teams to rely on date‑windowed filtering or custom client logic.

Does the API guarantee a stable ordering under concurrent writes? Can duplicate or skipped entries be observed when paging through actively updating dag runs? What client‑side strategies mitigate instability without relying on cursor‑based pagination?

Tasadduq Burney · Answer

AnswerNo. Airflow’s stable REST API v2.0+ does not guarantee stable, exactly-once ordering for limit/offset pagination when dag runs are created concurrently with paging. Duplicate or skipped entries can be observed.Confirmed facts: The stable API uses offset/limit pagination with a server default limit of 100 and a configurable maximum, typically 1000 via api.max_page_limit. Ordering is controlled by order_by, commonly execution_date descending, but the API does not provide snapshot isolation or a cursor token. The UI and CLI apply their own bounding limits separate from the API.Likely explanation for duplicates / missesUnder concurrent writes the result set shifts between requests. With offset-based paging:If new dag runs are inserted with a sort key that places them before the current page, the offset for the next request points past items that moved down, causing duplicates.If new dag runs are inserted with a sort key that places them after the current page but before the next offset window, items can be skipped.This is a property of offset pagination without a stable snapshot, not a bug in Airflow. Scheduler activity, dynamic DAGs, timetable triggers and backfills can create dag runs while a client is paging.Client-side mitigations without cursor paginationPage over a fixed date window. Use start_date_gte / start_date_lte or end_date_gte / end_date_lte with UTC timestamps that match Airflow storage. Keep the window immutable for the duration of paging.Fix ordering explicitly. Pass a deterministic order_by, e.g. -execution_date or execution_date, and avoid ordering by non-unique fields alone.Deduplicate by dag_run_id in the client. Collect seen IDs across pages and drop repeats; track the minimum/maximum execution_date seen to detect gaps.Reduce page size and latency. Smaller limit values reduce the time window for intervening inserts.Assumptions: Airflow 2.0+ stable REST API, endpoint such as GET /api/v2/dags/{dag_id}/dagRuns with limit/offset parameters. Behavior varies by version: defaults and max_page_limit changed across 2.0-2.4 and were standardized in 2.5+ OpenAPI spec. Database transaction isolation is READ COMMITTED by default, so newly committed runs become visible mid-page.One diagnostic detail that changes the recommendation: which Airflow version and which list endpoint you are paging, and whether api.max_page_limit has been changed from the default. Confirm with `airflow version` and GET /api/v2/config for api.max_page_limit.

Duplicate or missing entries when paginating Airflow REST API with limit/offset

1 answer

Answer

Likely explanation for duplicates / misses

Client-side mitigations without cursor pagination

Your perspective belongs here.

0 question comments