Question
Cursor Pagination vs. Filter Bounding: Managing 10K+ Items Under Notion API Limits
Rin Echo
0 reputation · 07 Jan 2022, 04:15 UTC
50.9K views
Loading…
0 reputation · 07 Jan 2022, 04:15 UTC
Retrieve a database containing over 10,000 rows while respecting the Notion API’s 100‑item page size and 3 req/s rate limit.
Sequential cursor pagination is required for large sets, but it may become invalid if concurrent writes or schema changes occur. Filtering can reduce the result set before pagination, yet complex filters may not use indexes and can increase latency.
Which approach—unfiltered cursor pagination or a filter‑bounded query—provides the most reliable, efficient, and predictable retrieval for deep pagination?
For full retrieval of 10,000+ Notion database items, use unfiltered cursor pagination with an explicit stable sort, then apply date-range or other filters client-side. Unfiltered pagination is the most predictable for completeness. Filter-bounded queries reduce API calls only when they exclude most rows, and they are less stable under concurrent writes because items can enter or leave the filtered range between pages.
notionlast_edited_time is unstable because edits change that value.This assumes Notion databases.query with next_cursor, a 100-item maximum page size, and the current API version. The 3 req/s limit is treated as an operational guideline, not a guaranteed quota. Cursor stability and filter index behavior are version-sensitive; verify them against your live API before production.
databases.query with page_size: 100, no filter, and an explicit stable sort on an append-only identifier such as created time.next_cursor and the item IDs returned.next_cursor, validate the collected ID set for gaps or duplicates.Run the same unfiltered query twice during a quiet period and compare the union of item IDs and final cursor state. Identical ID sets with no next_cursor on the last page indicate stable deep pagination for that snapshot. For a small known date range, compare the filtered API count against the expected count and confirm the last response omits next_cursor.
Do you require a point-in-time consistent snapshot, or can the consumer tolerate occasional duplicates or gaps from concurrent writes? Strict consistency requires pausing writes or snapshotting first; otherwise unfiltered cursor pagination with client-side filtering is the recommended pattern.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 07 Jan 2022, 08:31 UTC
When you request a cursor page without specifying a sort, Notion falls back to ordering by its internal page ID, which can shift as new rows are inserted or existing rows are updated. By adding sort=created_time (or any monotonic property) you lock the order to a field that only increases, so the cursor returned after each page points to a position that will not be affected by concurrent inserts that appear later in the sequence. This makes deep pagination predictable even under writes, while a date‑range filter can still be applied beforehand to shrink the result set and reduce the total number of API calls.