Stable REST API list endpoint returns fewer items than the requested limit

Answer first

Your export is truncating because Airflow REST API v1 silently clamps any client-supplied limit to the [api] maximum_page_limit value in airflow.cfg. When you request a limit larger than the cap, the endpoint returns fewer items than requested with no error, and your loop treats that short page as end-of-data.

Confirmed facts

• Airflow 2.x Stable REST API v1 caps the limit to maximum_page_limit and does so silently. The response contains a smaller items array and no error.
• total_entries is computed at query time per request. It is not stable for mutable tables. DAG runs and task instances can be created or updated between pages, so total_entries drifts.
• Airflow 3.x replaces the Flask-RESTful v1 API with FastAPI /api/v2. Pagination defaults, limit enforcement and response shape change, so v1 behavior does not transfer to 3.x.

Likely explanation

• The clamp is applied centrally by the API pagination helper used by /api/v1/dags/{dag_id}/dagRuns and /api/v1/dags/{dag_id}/taskInstances. In practice the silent clamp is consistent across those list endpoints in Airflow 2.x, though exact enforcement points and defaults are version-sensitive between 2.2-2.10.
• Offset paging over mutable data can skip or duplicate rows if inserts/deletes happen between calls. Relying on total_entries once and stepping offset by an assumed page size is not safe.

Steps for this case

1. Verify the effective cap. Issue a request with limit larger than you expect, e.g. limit=10000, and compare the returned items length to the request. Inspect airflow.cfg [api] maximum_page_limit on the server.
2. Use a page size that is <= the effective maximum. Do not request larger limits. Never treat a short page as end-of-data unless len(items) < effective_limit.
3. Make pagination deterministic. Add a stable order_by on immutable columns, e.g. execution_date and run_id for dagRuns, and a comparable stable ordering for taskInstances. Terminate only on an empty page, not on offset >= total_entries.
4. For a consistent export over mutable tables, prefer time-window slicing over pure offset. Page within a fixed execution_date window and advance the window after each complete pass, rather than relying on a global offset.

One diagnostic detail that changes the recommendation: what Airflow minor version are you running and what is the current [api] maximum_page_limit value? The default and enforcement point vary across 2.2-2.10, and the safe page size and response fields differ for /api/v2 in Airflow 3.x.