Deep Pagination in Elasticsearch: Choosing search_after Over from/size for Large Data Sets

The problem: hitting the 10,000‑result ceiling

Elasticsearch’s default index.max_result_window is 10,000. When a client requests from=15000&size=10 the coordinating node must ask every shard to collect the top 15,010 hits, sort them globally, and then discard the first 15,000. That O(N) work per shard blows heap and CPU, so the request fails with Result window is too large. Teams that need to page through millions of documents — log exports, audit trails, or ML feature extraction — quickly run into this wall.

Why from/size does not scale

• Memory pressure: each shard builds a priority queue of size from + size. Deep pages mean huge queues on every shard.
• CPU cost: global merge sort across shards repeats for every page.
• Hard limit: raising max_result_window above 50 k is possible but rarely safe; the coordinating node can OOM under load.

For a UI that only shows the first few pages, from/size is fine. For batch jobs that must walk the entire result set, it is the wrong tool.

search_after: a live cursor with constant per‑page cost

search_after avoids the global top‑N collection. You provide the sort values of the last hit from the previous page, and each shard returns the next slice. The cost stays O(page size) per shard no matter how deep you go.

Deterministic sort is mandatory

Because the cursor is built from sort values, duplicate sort keys cause missed or duplicated documents. The standard fix is a compound sort that ends with a unique tie‑breaker — usually _id:

GET /logs/_search
{
  "size": 100,
  "sort": [
    { "@timestamp": { "order": "asc" } },
    { "_id": { "order": "asc" } }
  ]
}

Every subsequent request copies the sort array of the last hit into search_after:

GET /logs/_search
{
  "size": 100,
  "sort": [
    { "@timestamp": { "order": "asc" } },
    { "_id": { "order": "asc" } }
  ],
  "search_after": [ "2024-06-15T12:34:56.789Z", "abc123" ]
}

Run these requests from Kibana Dev Tools, curl, or any Elasticsearch client. You need read permission on the target index (or monitor on the cluster for admin scripts).

Worked example: paging 50 k log lines with a consistent snapshot

Below is a minimal, reproducible flow. It assumes Elasticsearch 8.x and a user with manage_index and write rights on the demo_logs index.

1. Create the index with a keyword _id and a date field.

   PUT /demo_logs
   {
     "settings": { "number_of_shards": 2 },
     "mappings": {
       "properties": {
         "@timestamp": { "type": "date" },
         "message": { "type": "keyword" }
       }
     }
   }

2. Bulk‑index 50 000 documents (scripted in your language of choice). Each doc gets a unique _id and a sequential timestamp.
3. Open a point‑in‑time (PIT) to freeze the view.

   POST /demo_logs/_pit?keep_alive=2m
   # → { "id": "", "keep_alive": "2m" }

   Keep the returned id for the whole pagination run.

4. First page using the PIT.

   POST /_search
   {
     "size": 100,
     "pit": { "id": "", "keep_alive": "2m" },
     "sort": [
       { "@timestamp": { "order": "asc" } },
       { "_id": { "order": "asc" } }
     ]
   }

5. Loop: take the sort array of the last hit, feed it to search_after, repeat until hits.total.value equals the number of returned docs.

   POST /_search
   {
     "size": 100,
     "pit": { "id": "", "keep_alive": "2m" },
     "sort": [
       { "@timestamp": { "order": "asc" } },
       { "_id": { "order": "asc" } }
     ],
     "search_after": [ "2024-06-15T12:34:56.789Z", "doc_00123" ]
   }

6. Delete the PIT when done.

   DELETE /_pit
   {
     "id": ""
   }

Verification: Count the total docs returned across all pages; it must equal 50 000. Insert a new document between two pages while the PIT is alive — it will not appear, proving snapshot isolation. Without the PIT the same insert would surface on the next page.

Trade‑offs and limitations

Aspect	from/size	search_after (no PIT)	search_after + PIT
Maximum depth	10 k (default)	Unbounded	Unbounded
Per‑page latency	Grows with page number	Stable	Stable + small PIT overhead
Real‑time consistency	Live view	Live view (new docs appear)	Snapshot (new docs hidden)
Jump‑to‑page UI	Supported	Not supported	Not supported
Resource retention	None	None	Segment files kept until PIT expires

• Shard count matters: every search_after request fans out to all shards. Fewer, larger shards or routing the query to a subset reduces latency.
• Sort field updates: if a document’s @timestamp changes between pages, it can jump forward or disappear because the cursor follows sort values, not IDs.
• PIT cleanup: forgetting to delete a PIT leaks resources until keep_alive expires. Automate deletion in a finally block or use a short keep_alive (e.g., 2 m) and refresh it each request.
• Runtime fields in sort: they require doc_values or fielddata. Sorting on a text field without a keyword sub‑field triggers fielddata loading and can OOM.

When to reach for composite aggregations

If you need to paginate aggregation buckets (e.g., distinct user IDs for a funnel) rather than raw hits, use a composite aggregation with after_key. It shares the same cursor semantics but works on the aggregation layer, avoiding hit‑level pagination entirely.

Actionable checklist

1. Default to search_after with a deterministic [timestamp, _id] sort for any batch job that may exceed 10 k results.
2. Add a PIT only when you need a repeatable snapshot (exports, audits). Keep keep_alive short and delete explicitly.
3. Monitor per‑page latency in the slow log; a sudden spike often signals a shard‑count or heap issue.
4. Reserve from/size for the first few pages of a user‑facing UI; never use it for deep scrolling.
5. Test the full pagination path in a staging cluster with a realistic document count before shipping.

By treating pagination as a cursor problem instead of an offset problem you avoid the coordinating‑node memory spike, get predictable latency, and — when you need it — a clean point‑in‑time view without the heavy segment retention of the legacy scroll API.