Persistent pagination: moving from OFFSET pages to keyset cursors for large tables

A Haskell service using the persistent library needs to page through a table that can grow to millions of rows. The straightforward approach — selectList with LimitTo and OffsetBy, which maps to SQL LIMIT/OFFSET — has two drawbacks at scale: the database still reads and discards every row preceding the offset, and rows inserted or deleted between requests can make a page skip or repeat records.

Keyset (cursor) pagination — ordering by a unique key and filtering on a last-seen value — avoids both problems, but persistent does not ship a unified, backend-agnostic cursor pagination API, so the predicate and cursor state must be assembled per query. A second boundary is how pages are consumed: a plain list per request versus a streaming abstraction such as conduit, which changes memory residency for very large result sets.

For a table with a monotonic primary key, how much of a keyset predicate — especially one that must also tie-break on a secondary column — can be expressed with persistent's filter and order combinators before raw SQL is required? And once pages are produced, is conduit-based streaming worth the added dependency compared with bounded list pages?