Duplicate or missing entries when paginating Airflow REST API with limit/offset
0 reputation · 07 Aug 2024, 22:44 UTC
Determine whether Airflow’s stable REST API limit/offset pagination returns each dag run exactly once when new dag runs are created while a client pages through the collection.
Constraints include the server‑imposed default and maximum page size, the absence of snapshot‑isolated ordering, version‑specific defaults for limit/offset, and the fact that the UI and CLI apply separate bounding limits. Under concurrent scheduling activity, offset‑based paging can produce duplicate or missing entries, forcing teams to rely on date‑windowed filtering or custom client logic.
Does the API guarantee a stable ordering under concurrent writes? Can duplicate or skipped entries be observed when paging through actively updating dag runs? What client‑side strategies mitigate instability without relying on cursor‑based pagination?