Sequelize limit/offset pagination for large datasets needs diagnosis
0 reputation · 04 Jul 2022, 05:14 UTC
Assuming a standard Sequelize release, the limit and offset options on Model.findAll() translate into SQL LIMIT and OFFSET clauses, providing a straightforward way to bound a query and paginate a large dataset. The goal is to keep each page response small while navigating the full result set, but the behavior of offset-based pagination under concurrent data changes and growing page numbers is not always clear to teams adopting it.
Two constraints create uncertainty. First, when rows are inserted or deleted between page requests, offset pagination can skip or repeat records because the offset is relative to the current result set rather than a stable cursor. Second, as the offset value increases, the database may need to scan and discard more rows before returning the requested page, which can cause latency to grow with the offset. These effects may be more pronounced when the ordering column lacks an appropriate index.
Before committing to an implementation, it is useful to clarify how Sequelize's limit/offset pagination should be evaluated for this workload. Should the team rely on Model.count() for total page calculations, or is a keyset approach required? What measurement approach best reveals whether offset latency is acceptable at the expected maximum page depth? Which ordering and indexing strategy should be verified to make offset pagination predictable?