Offset vs Keyset Pagination for Soft‑Deleted Records in NestJS: Which Keeps Page Size Consistent?
0 reputation · 01 Jan 2022, 14:59 UTC
0 reputation · 01 Jan 2022, 14:59 UTC
When building a NestJS API that serves large datasets, developers must decide how to paginate queries while respecting soft‑deleted records. Offset‑based pagination (limit/offset) is easy to add with QueryPipe and TypeORM’s findAndCount, but if the total count includes soft‑deleted rows, the number of active entities returned on a page can fall below the requested limit. Keyset (seek) pagination avoids OFFSET by using a cursor on a unique, sortable column, yet applying a WHERE clause for deletedAt IS NULL can shift the cursor and cause gaps or duplicate rows when deletions occur.
The goal is to choose a pagination strategy that returns exactly the requested number of active items per page, regardless of interleaved soft‑deleted rows, without sacrificing performance or complicating client‑side navigation. Which approach—adjusting the offset to skip deleted rows, filtering before pagination, or adapting the keyset cursor to ignore deleted IDs—best satisfies this requirement, and what trade‑offs arise in implementation complexity and query efficiency?
35275 reputation · 01 Jan 2022, 17:07 UTC
To maintain a consistent page size when dealing with soft-deleted records, keyset pagination (seek method) is the correct choice. Unlike offset pagination, which counts rows regardless of their visibility status, keyset pagination filters the dataset first and then fetches the next N available records based on a unique cursor, ensuring the requested limit is always met.
Offset pagination relies on the OFFSET clause in SQL. If your query logic applies a filter for soft-deleted records (e.g., WHERE deletedAt IS NULL) after the database calculates the offset, or if the offset is calculated against the total table count including deleted rows, the resulting page size will fluctuate. Even when filtered correctly, OFFSET remains inefficient for large datasets because the database must scan and discard all preceding rows before returning the requested page.
Keyset pagination avoids the OFFSET keyword entirely. Instead, it uses the value of the last item from the previous page as a pointer (the cursor) for the next query.
id or a precise createdAt timestamp).// Example TypeORM implementation in a NestJS service
async paginate(cursor: number, limit: number) {
const records = await this.repository.find({
where: {
id: MoreThan(cursor),
deletedAt: IsNull(),
},
order: { id: 'ASC' },
take: limit,
});
const nextCursor = records.length > 0
? records[records.length - 1].id
: null;
return { data: records, nextCursor };
}
| Feature | Offset Pagination | Keyset Pagination |
|---|---|---|
| Page Size | Inconsistent (if not carefully filtered) | Consistent |
| Performance | Degrades as offset increases (O(n)) | Constant performance (O(log n)) |
| Navigation | Supports random access (Page 5) | Sequential only (Next/Previous) |
To verify this behavior, enable the TypeORM logger. Ensure the generated SQL for keyset pagination uses a WHERE clause for both the cursor and the deletedAt column, and contains no OFFSET clause. Test with a dataset where every second record is soft-deleted to confirm the page size remains exactly equal to the limit.
Missing Diagnostic Detail: Is your ordering column guaranteed to be unique? If you are sorting by a non-unique field (like category), you must implement a tie-breaker (e.g., ORDER BY category ASC, id ASC) to prevent records from being skipped.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 02 Jan 2022, 02:20 UTC
When soft-deleted rows coexist with active records, offset pagination often returns fewer items than the requested limit because the count query includes invisible rows while the data query discards them. A practical middle ground is to materialize an active-only subquery or view once, then apply keyset pagination on its monotonically increasing cursor. This guarantees exactly N active records per page with predictable seek performance, avoiding the growing scan cost of OFFSET. The trade-off is an additional schema layer and the need to refresh the view or recompute the cursor after each bulk soft-delete. For most NestJS APIs with frequent reads and occasional writes, this hybrid approach stabilizes client-side pagination without the complexity of regenerating cursors after every deletion.