Direct Answer
The intermittent timeouts occur because GROQ slice pagination order(field)[offset...limit] forces the Content Lake backend to materialize and sort the entire matching result set before returning the requested slice. Cost grows linearly with offset depth and result-set size, so deep pages hit query-complexity thresholds or the hard per-request document cap, triggering 400-series errors or genuine timeouts.
Likely Explanation vs. Confirmed Facts
- Confirmed: Slice syntax requires a full sort pass; ordering by non-indexed or high-cardinality computed fields is substantially slower than indexed system fields (
_createdAt, _id).
- Confirmed: When the sort key has duplicate values, slice boundaries are not stable across requests—documents sharing the same key can shift between pages unless a tiebreaker (
order(field, _id)) is added.
- Confirmed: The API enforces a maximum documents-per-request limit and rejects overly expensive queries with 400-series errors; very large offsets effectively hit the same wall.
- Version/plan-dependent: Exact per-request caps, query-cost thresholds, and timeout durations are not publicly fixed—verify against your current Sanity plan documentation.
Recommended Mitigation Steps
- Switch to cursor/keyset pagination for deep traversal:
// Instead of: *[_type == "post"] | order(publishedAt)[50000...50100]
// Use:
*[_type == "post" && publishedAt > $lastSeenPublishedAt && _id > $lastSeenId]
| order(publishedAt, _id) [0...100]
This keeps per-request work roughly constant regardless of depth.
- Add a tiebreaker to any
order() clause: order(publishedAt, _id) guarantees stable page boundaries when publishedAt has duplicates.
- Reduce per-request cost: project only needed fields, narrow the filter before ordering, and keep page sizes modest (50–200 docs).
- Reserve offset pagination for shallow UI cases (first 3–5 pages) where the full sort cost is acceptable.
One Diagnostic Detail That Changes the Recommendation
Run the same query at increasing offsets (e.g., 0, 5,000, 20,000) and log response time, HTTP status, and any x-query-cost or error-body limit messages. If you see 400-series errors with cost-limit text, you’re hitting a hard query-complexity ceiling—keyset pagination is mandatory. If you see genuine 504/timeout responses without cost warnings, the dataset size or sort-field cardinality may require further filter narrowing or a smaller page size in addition to keyset pagination.
sanity