GROQ Pagination: Transitioning from Unbounded Queries to Sliced Result Sets
26.5K reputation · 25 Jun 2023, 00:30 UTC
Managing Large Datasets in Sanity Content Lake
When scaling a Sanity.io project, retrieving large document collections via the GROQ API requires a shift from unbounded queries to a bounded pagination strategy. The API enforces limits on the number of documents returned per request to maintain performance and prevent timeouts.
Implementing pagination typically involves the slice operator [start...end] to request specific windows of data. However, because this mechanism is stateless, the client must manage the offset indices and potentially execute a separate count() query to determine the total available records for the UI.
As the offset increases during deep pagination, there is a known risk of increased latency since the engine must scan preceding records before returning the requested slice.
- How does the performance of the slice operator scale as the offset index reaches very high values in large datasets?
- What is the recommended strategy to mitigate latency during deep pagination without relying on client-side caching?
sanity1 answer
1 question comment
Use comments to ask for clarification. Post a solution as an answer.
1,850 reputation · 25 Jun 2023, 07:23 UTC
While the slice operator [start...end] is evaluated after ordering and filtering, Sanity automatically creates indexes on _updatedAt, _createdAt, and any field used in an order() clause. This index lets the engine locate the first matching document faster, but it still must walk through the ordered index to skip start entries, so latency grows roughly linearly with the offset. To keep pagination stable when multiple documents share the same sort value, extend the order with a tie‑breaker such as _id (e.g., order(publishedAt desc, _id asc)).