ActiveRecord and PostgreSQL: Offset pagination vs keyset pagination for large datasets under concurrent writes
0 reputation · 07 May 2020, 02:46 UTC
Goal: Implement pagination for a table exceeding 100 k rows while the application experiences high insert/update concurrency, ensuring that users never see skipped or duplicate records when navigating pages.
Constraints: Standard OFFSET/LIMIT becomes costly on deep pages and can return inconsistent results under concurrent writes. Keyset (cursor) pagination avoids the scan penalty and provides stable ordering, but requires a unique, immutable sort column or a composite cursor (e.g., id + created_at), which adds client‑side complexity. ActiveRecord offers find_each/batches for background iteration but lacks a built‑in cursor API, and popular gems like Kaminari default to OFFSET.
Uncertainty: Whether to adopt cursor‑based pagination as the default strategy, retain OFFSET for its simplicity and deep‑linking capability, or hybridize the two approaches.
Questions:
- What patterns or extensions can be used to implement keyset pagination in ActiveRecord without sacrificing the convenience of existing pagination helpers?
- How should composite cursors be handled when the primary sort column is not unique, and what impact does this have on API contracts?
- Is there a recommended way to combine OFFSET for shallow pages with cursor pagination for deep pages within a single endpoint?