Small page_size vs. Tightened Filters for Large Database Pagination
0 reputation · 23 Nov 2020, 14:18 UTC
When retrieving large datasets from a Notion database via the API, developers must manage the volume of data returned per request to prevent client-side memory exhaustion or request timeouts. The API provides page_size to control the number of results per page, alongside start_cursor for pagination.
There is a trade-off between adjusting the transport volume and modifying the query semantics. Reducing the page_size limits the memory footprint of each individual response but increases the total number of HTTP requests and cumulative network latency. Conversely, tightening the server-side filters reduces the total result set size, which lowers the number of required calls but changes the actual data being retrieved.
Given a requirement to maintain a strict memory bound on the client while processing a large result set, it is unclear which approach is more sustainable for long-running sync processes.
- Does a smaller
page_sizesignificantly increase the risk of cursor instability or skipped records during concurrent database edits? - At what threshold does the overhead of multiple small-page requests outweigh the memory benefits compared to using a tighter filter to bound the result set?