Limit and Offset Pagination in Racket DB: Unresolved Streaming API Gap

Pagination with LIMIT/OFFSET

The Racket DB library’s query-values function supports SQL LIMIT and OFFSET clauses, allowing developers to fetch a specific slice of a large table. The goal is to reduce memory consumption and improve response times when handling datasets that exceed available RAM.

However, query-values and query-rows eagerly collect all rows that match the query into a list before returning control to the caller. This eager collection becomes a bottleneck when the result set is very large, even if the LIMIT is small. Moreover, using very large OFFSET values can degrade performance on many SQL backends because the database must scan and discard rows before returning the desired page.

There is currently no documented, stable API in the Racket DB library that streams rows incrementally or exposes server‑side cursors for large result sets. Community discussions have suggested potential extensions, but no official implementation exists yet.

Unresolved Questions

• Does the Racket DB library provide or plan to provide a streaming or cursor‑based API for large result sets?
• What are the performance trade‑offs between using large OFFSET values and alternative pagination strategies such as key‑set pagination or manual cursor handling?
• Are there community‑maintained extensions or recommended patterns to handle very large tables without loading all rows into memory?