Direct Answer
RediSearch does not provide a native cursor, scroll, or seek API. The LIMIT offset count clause is applied after the result set is filtered and sorted, meaning the engine must materialize and skip offset number of documents before returning the requested page. This results in latency that grows linearly with the offset size. Furthermore, because RediSearch lacks snapshot isolation for queries, concurrent index mutations (inserts, deletes, or updates) can shift document rankings between requests, leading to duplicate or missing items when using offset-based pagination.
The Mechanics of Offset Performance
When executing a query like LIMIT 10000 10, RediSearch must identify and sort the first 10,010 matching documents to determine which ten occupy the requested window. This creates significant CPU and memory pressure on the Redis node as the offset increases. Because the application must manage the state of the pagination, any shift in the underlying data causes the offset to point to a different logical item than it did in the previous request.
Deterministic Strategy: Keyset Pagination
To achieve deterministic pagination and avoid the performance penalty of deep offsets, implement keyset pagination (also known as the seek method). This replaces the offset with a range filter on a stable, unique field.
- Requirement: Ensure the field used for pagination (e.g., a numeric
id or timestamp) is defined as SORTABLE in the index schema.
- Initial Request: Fetch the first page using
LIMIT 0 10 SORTBY id ASC.
- Subsequent Requests: Capture the value of the last item on the current page (
last_id) and use it as a filter for the next page:
FT.SEARCH idx "query @id:[(last_id +inf]" LIMIT 0 10 SORTBY id ASC
This approach ensures that the engine jumps directly to the starting point of the next page, maintaining performance regardless of depth and ensuring consistency even if documents are added to the beginning of the result set.
Handling Relevance Sorting
If sorting by relevance (BM25), scores are not guaranteed to be unique. To prevent non-deterministic ordering, always include a unique tie-breaker in the SORTBY clause:
FT.SEARCH idx "query" LIMIT 0 10 SORTBY score DESC id ASC
Verification and Monitoring
To validate the performance impact of your pagination strategy, use the following commands:
- Profile Deep Offsets: Run
FT.PROFILE SEARCH idx "query" LIMIT 10000 10 to observe the internal scan count and execution time.
- Verify Schema: Use
FT.INFO idx to confirm that your pagination field is marked as SORTABLE.
- Test Consistency: Perform a
LIMIT 0 20 query, note the 11th ID, and then perform a LIMIT 10 10 query to verify the first returned ID matches the 11th from the first set on a static index.
Missing Diagnostic Detail
To refine this recommendation: Which version of RediSearch are you using, and is the field you intend to use for pagination already defined as SORTABLE in your schema?
redis