Limit and Offset Behavior in Knex.js: Unresolved Pagination API Decision
0 reputation · 31 Mar 2024, 14:31 UTC
0 reputation · 31 Mar 2024, 14:31 UTC
To bound large result sets efficiently in Knex.js using the documented .limit() and .offset() methods while avoiding performance pitfalls.
Large OFFSET values force the database to scan preceding rows, causing noticeable latency on MySQL and PostgreSQL. Knex currently requires developers to manually chain .limit() and .offset() and issue separate COUNT(*) queries to compute total pages. No native .paginate() helper exists, and community discussions have not reached consensus on its design.
.paginate() helper that automatically returns both the page slice and total record count?OFFSET values?32775 reputation · 01 Apr 2024, 00:29 UTC
Knex.js should not add a magic .paginate() to core. Keep .limit() and .offset() as low-level primitives, and provide pagination as an opt-in helper or plugin that composes them. A built-in helper that always returns a total count would couple the query builder to expensive, racy COUNT(*) semantics that many applications do not need.
.limit(n) and .offset(n) and emit dialect-appropriate SQL. There is no core method that returns both the page slice and a total record count.LIMIT/OFFSET returns only the slice, totals require a separate COUNT(*) query or a window function. Under concurrent writes, the row set and the total can be inconsistent unless both run inside one transaction with a consistent snapshot..limit(n).offset(m) for bounded slices when deep paging is rare.COUNT(*) in the same transaction so the numbers agree.SELECT * FROM items
WHERE (sort_key, id) < (:last_key, :last_id)
ORDER BY sort_key DESC, id DESC
LIMIT :n;hasMore by requesting n + 1 rows and discarding the extra, avoiding COUNT(*) entirely..toSQL() and run EXPLAIN/ANALYZE on your target dialect before shipping.Use explicit parameters—limit/offset, page/perPage, or cursor/after—and never silently clamp or mutate inputs. A practical return shape is:
{
data: [...],
pagination: {
limit, offset, page, perPage,
total, totalPages, hasMore, nextCursor
}
}
Compute total and totalPages only when requested. Validate non-negative integers, enforce a maximum limit, reject unsupported dialect combinations, and fail clearly when a cursor is requested without a usable sort key.
The single missing diagnostic that alters the recommendation is whether your consumers need exact total record counts or only a hasMore signal. Admin UIs with page numbers may justify transactional COUNT(*); infinite-scroll consumers should omit it by default.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.