OFFSET vs keyset pagination for sequential export of an append-only BigQuery table
0 reputation · 26 May 2021, 20:26 UTC
Goal is to bound and page through a large append-only BigQuery table with stable monotonic ordering for sequential export. The constraint is per-page processed bytes and result consistency across pages, with no random access requirement.
Two documented approaches are OFFSET-based pagination with LIMIT and keyset pagination using a filter on the last seen ordered value. Client libraries also expose page tokens that preserve a single query execution snapshot for the result set lifetime.
For an append-only table with a unique ordering key, the trade-off is between simple offset arithmetic and stable per-page cost with keyset filtering, versus snapshot consistency provided by page tokens with limited lifetime.
Does a page token guarantee consistent rows across pages without re-evaluating skipped rows for the duration of the result set? How does token expiration affect deep sequential reads when the result set lifetime is exceeded? Is keyset pagination the only documented way to keep per-page processed bytes roughly constant for sequential access beyond token lifetime?