Elasticsearch Deep Pagination: from/size vs search_after

The decision: paginate past 10,000 documents without destabilizing the cluster

Your application needs to return results beyond the first 10,000 documents in an Elasticsearch index. The default index.max_result_window setting is 10,000. Queries that exceed it with from and size receive a 400 error because the result window—the number of top hits Elasticsearch tracks per query—has been exhausted.

Useful takeaway: Keep from/size for shallow, user-facing pages inside the default window. Use search_after for exports, infinite feeds, dashboards, or any client that must page through large result sets. search_after is the supported mechanism for deep pagination and does not require raising index.max_result_window.

This guide assumes Elasticsearch 7.x or 8.x. Behavior depends on cluster version, distribution, and index settings such as refresh_interval, which defaults to 1 second and controls how soon new documents become visible.

Constraints to set before choosing

• Result depth: how many documents the client must traverse.
• Access pattern: random page jumps versus sequential scrolling.
• Sort requirements: whether results need a stable, deterministic order.
• Cluster headroom: available heap—the JVM memory used by Elasticsearch—and garbage-collection (GC) tolerance.

Supported options compared

Option	Best for	Limit	Server state	Client requirement
from / size	Shallow user-facing pages within the result window	Bounded by index.max_result_window, default 10,000	None beyond normal query coordination across shards	Page number and page size
search_after	Deep pagination, exports, infinite scrolling	No hard result-window cap	No retained scroll context; cursor is client-side	Last sort values from previous page and a deterministic sort

Trade-offs

from / size

Simple to implement: specify from as the offset and size as the page size. Elasticsearch coordinates the request across shards—partitions of an index—sorts the top from + size matches, and discards earlier ones. As from grows, each page costs more heap and CPU because the cluster must process and merge increasingly large intermediate result sets.

Raising index.max_result_window to allow deeper from paging is possible, but it increases heap pressure, GC pauses, and shard-coordination overhead. It is not a recommended path for production deep pagination.

search_after

search_after uses the sort values of the last document on the current page to locate the next batch. Elasticsearch does not retain a scroll context—server-side search state—so there is no server-side cursor to clean up. The client holds the sort-state and sends it with every request.

The trade-off is sequential access only: you cannot jump directly to page 50. You must iterate from the beginning or store bookmarks for specific sort positions. Results also reflect near-real-time visibility; documents indexed within the last refresh interval may not appear yet.

Implementation: search_after with a deterministic sort

search_after requires a sort order that cannot tie. If two documents share the same sort value, Elasticsearch cannot reliably decide which comes next, and pages can skip or repeat documents. Add a unique tie-breaker as the final sort key, such as a document ID field or _doc, which sorts by internal Lucene document order.

Run the following in a terminal with network access to the cluster. You need read permission on the target index. Replace placeholders in angle brackets.

# First page: no search_after
curl -s -X GET "<CLUSTER_URL>/<INDEX_NAME>/_search" \
  -H 'Content-Type: application/json' \
  -d '{
    "size": 100,
    "sort": [
      {"@timestamp": {"order": "asc"}},
      {"_doc": {"order": "asc"}}
    ],
    "query": {
      "match_all": {}
    }
  }'

Extract the sort array from the last hit in the response, then pass it as search_after on the next request:

# Subsequent page: replace <SORT_VALUES> with the last hit's sort array
curl -s -X GET "<CLUSTER_URL>/<INDEX_NAME>/_search" \
  -H 'Content-Type: application/json' \
  -d '{
    "size": 100,
    "search_after": <SORT_VALUES>,
    "sort": [
      {"@timestamp": {"order": "asc"}},
      {"_doc": {"order": "asc"}}
    ],
    "query": {
      "match_all": {}
    }
  }'

_doc is efficient but not stable across index updates or reindexing. For a stable tie-breaker across writes, use a unique indexed field such as a UUID or primary key instead.

Validation and diagnostic checks

1. Confirm the result-window boundary. Request from=9990 and size=20 against an index with more than 10,000 matching documents. Expect a 400 error mentioning result_window unless index.max_result_window has been raised.
2. Run the same depth with search_after. Paginate sequentially using the last sort values from each response. Confirm each batch returns without a result-window error.
3. Check sort determinism. Compare consecutive pages for missing or duplicate documents. If either appears, the sort lacks a unique tie-breaker.
4. Monitor cluster cost. Check node heap and garbage-collection logs during a deep pagination run. search_after should avoid the per-request merge cost that grows with from × size.

Limitations

• search_after does not support jumping to an arbitrary page number.
• It is not a point-in-time snapshot—a stable view of the index at one moment—unless combined with a version-appropriate point-in-time API. Documents updated or deleted between pages can shift results.
• Sorting on text or analyzed fields is unreliable. Use keyword, numeric, date, or _doc with a unique field for deterministic ordering.
• Performance can degrade with very high sort cardinality or many shards. Test with production-shaped data and shard counts before adopting it at scale.

Practical way to confirm the result

After implementing search_after, count documents returned across ten consecutive pages and compare that count against a direct count query for the same filter. If the totals match and no document appears twice, the pagination is deterministic and complete for the current index state.

Because this guide recommends search_after instead of changing index.max_result_window, no state-changing rollback is required.