Choosing Elasticsearch Pagination: from/size, search_after, or Scroll

The Pagination Problem

Retrieving large, ordered result sets from Elasticsearch requires choosing a pagination mechanism that matches your latency, consistency, and resource constraints. The wrong choice causes response times to grow linearly with offset, triggers circuit breakers, or returns duplicates and skipped documents as the index changes.

This guide assumes Elasticsearch 7.x+ semantics. Point in Time (PIT) and search_after parameter names and behavior differ across major releases, so verify your cluster version before implementing.

Decision Constraints

• Latency: maximum acceptable response time per page.
• Consistency: whether results must remain stable across pages while the index receives updates.
• Depth: whether users browse the first few pages or skip to very deep offsets.
• Resource budget: available heap and tolerance for circuit breaker trips.

Supported Options Comparison

Mechanism	Logic	Latency profile	Consistency	Resource impact	Best case
from/size	Offset-based collection. The cluster gathers from + size hits across shards and discards the first from.	Constant for shallow pages; grows linearly with offset.	Low. Index updates shift results between pages.	High for deep offsets. Can trigger the circuit breaker.	UI first pages under roughly 10,000 hits
search_after with PIT	Cursor-based pagination. A Point in Time freezes a search context; search_after resumes from the last sort values.	Bounded and roughly constant per page regardless of depth.	High within the PIT session. Later updates do not change the current view.	Low per request. PIT holds segment references until closed or expired.	Deep UI browsing and ordered feeds
Scroll API	Server-side cursor that materializes a fixed snapshot for bulk retrieval.	High throughput for large batches.	Absolute snapshot. No changes are visible after the scroll opens.	High. Retains segment references and prevents merges until cleared or timed out.	Bulk export and reindexing

Trade-offs

from/size at depth

A request such as from: 50000, size: 10 forces every shard to collect and sort 50,010 documents before returning the final ten. The coordinating node merges these lists, consuming heap and increasing latency. Do not use this pattern for deep pagination in production.

search_after requires a deterministic sort

Sort values must uniquely identify a document. Sorting only by @timestamp can cause duplicates or skips when multiple documents share the same timestamp. Add a unique tie-breaker such as _id or a primary key field.

Scroll retention

Scroll holds old segment references, which blocks segment merging and can increase disk and heap usage. Always clear the scroll when the job finishes, and set a scroll timeout that covers a single batch, not the entire export.

Implementation: Deep Pagination with PIT and search_after

Run these requests from a client with read access to the target index. Replace <pit_id> and <sort_values> with values returned by the API.

1. Open a Point in Time

POST /my-index/_pit?keep_alive=1m
{}

Expected result: a JSON body containing an id field, which is the PIT identifier. Risk: if the client takes longer than keep_alive between requests, the PIT expires and subsequent searches fail.

2. Request the first page

POST /_search
{
  "size": 100,
  "pit": { "id": "<pit_id>", "keep_alive": "1m" },
  "sort": [ { "@timestamp": "desc" }, { "_id": "desc" } ],
  "_source": ["id", "@timestamp"]
}

Extract the sort array from the last hit in the response.

3. Request subsequent pages

POST /_search
{
  "size": 100,
  "pit": { "id": "<pit_id>", "keep_alive": "1m" },
  "sort": [ { "@timestamp": "desc" }, { "_id": "desc" } ],
  "search_after": [ "2026-10-17T21:00:00Z", "doc_123" ],
  "_source": ["id", "@timestamp"]
}

Pass the previous page's final sort values as search_after. Repeat until the response returns fewer hits than requested or an empty hits array.

4. Close the PIT

DELETE /_pit
{ "id": "<pit_id>" }

Closing releases segment references. If the client crashes, the PIT expires automatically after keep_alive.

Implementation: Bulk Export with Scroll

Use this pattern for offline export jobs, not interactive user traffic.

POST /my-index/_search?scroll=5m
{
  "size": 1000,
  "sort": ["_doc"]
}

Use the returned scroll_id to fetch the next batch. When the export finishes, call:

DELETE /_search/scroll
{ "scroll_id": ["<scroll_id>"] }

Leaving a scroll open retains resources until the timeout elapses.

Validation

• Latency test: index 1,000,000 documents with a sortable timestamp and unique id. Compare response time for from: 0 versus from: 50000, then measure the 500th page using search_after with PIT.
• Consistency check: while paginating with a PIT, update a document on a later page. Confirm the update does not appear in the current PIT session.
• Resource check: use the _nodes/stats API to observe heap usage and segment counts during a large Scroll job versus a PIT session.
• Profiling: add "profile": true to a search request and compare the collect phase cost for from/size versus search_after.

Limitations

• PIT and search_after behavior is version sensitive. Confirm parameter names for your Elasticsearch major version.
• from/size with large offsets can exhaust heap and trip the circuit breaker.
• search_after requires a deterministic sort; without a unique tie-breaker, concurrent updates can cause duplicates or skips.
• Scroll holds segment references and can cause memory pressure if scrolls are not cleared.

Practical check: after implementing search_after with PIT, verify that sort values returned across consecutive pages are monotonic for the chosen sort order and that no document id appears twice.