Deep Pagination in Elasticsearch: Why from/size Fails and How search_after + PIT Works

Page 200 returns an error, not an empty page

You build a results table with a next button. It works for the first few hundred rows. Then someone clicks deep enough and the API returns a 400 instead of data, and the message names index.max_result_window.

That setting — commonly 10,000 by default, but configurable per index and via templates — is not an arbitrary rate limit. It exists because from/size is an offset. To return hits 10,000–10,010, every shard has to build and sort its own top 10,010 candidates, the coordinating node merges them, and almost all of that work is discarded. Cost grows with depth, and the window is the guardrail.

The fix is not raising the window. It is replacing the offset with a cursor: search_after, anchored to a point-in-time (PIT) so the result set does not shift underneath you.

search_after replaces "skip N" with "resume after this document"

Instead of telling Elasticsearch how many documents to skip, you hand it the sort values of the last document you already received. The engine resumes from that position. Nothing is materialized for pages you never asked for, so page 500 costs roughly what page 1 costs.

Two conditions make this correct:

• A deterministic sort. If two documents share the same sort key, "after this document" is ambiguous and you can skip or repeat rows.
• A unique tiebreaker. Append one so the ordering is total.

Inside a PIT, the usual tiebreaker is _shard_doc, an implicit per-shard document order. Outside a PIT, _doc is not a stable global ordering, and sorting on _id is discouraged because it requires enabling fielddata on the _id field.

A point-in-time pins the view across requests

Pagination is multiple requests. Between them, refreshes, merges and deletes change what a query would return. A PIT freezes a consistent view of the indices for the life of the context, so a document deleted mid-walk does not vanish from page 7 and shift everything after it.

PITs hold cluster resources. Close them on every exit path, including error paths.

Worked example: reproduce the failure, then walk it with a PIT

Run these in Kibana Dev Tools or as curl against your cluster. You need read access to the target indices; confirm the exact privilege names for your version. Use a throwaway index.

First, create an index with a deliberately small window so the failure is cheap to reproduce:

PUT /events-deep-paging
{
  "settings": { "index.max_result_window": 100 },
  "mappings": {
    "properties": {
      "created_at": { "type": "date" },
      "message":    { "type": "keyword" }
    }
  }
}

Index a few hundred documents with _bulk or a small script, then try to read past the window:

GET /events-deep-paging/_search
{
  "from": 100,
  "size": 10,
  "sort": [ { "created_at": "asc" }, { "_doc": "asc" } ]
}

Expect a rejection whose message names the window limit and the configured value. That is the behavior you are working around, not a bug to file.

Now open a PIT:

POST /events-deep-paging/_pit?keep_alive=1m

The response contains an id. Call it <pit_id>. Note that PIT searches go to /_search with no index in the path — the PIT carries the index list.

GET /_search
{
  "size": 10,
  "query": { "match_all": {} },
  "sort": [ { "created_at": "asc" }, { "_shard_doc": "asc" } ],
  "pit": { "id": "<pit_id>", "keep_alive": "1m" }
}

Each hit now carries a sort array, for example [1759000000000, 42]. Copy the array from the last hit verbatim into the next request:

GET /_search
{
  "size": 10,
  "query": { "match_all": {} },
  "sort": [ { "created_at": "asc" }, { "_shard_doc": "asc" } ],
  "pit": { "id": "<pit_id>", "keep_alive": "1m" },
  "search_after": [1759000000000, 42]
}

Repeat until a response returns fewer hits than size. Then close the PIT:

DELETE /_pit
{ "id": "<pit_id>" }

Use the most recent id: a PIT search response can return an updated one, and closing a stale id may not release the context.

What you give up

• No random access. search_after is forward-only. Jump to page 47 is not available, so offer next/previous, infinite scroll, or a narrower filtered search instead.
• No free total count. track_total_hits defaults to a bounded value (commonly 10,000), so a label like showing N of M is a lower bound unless you raise it to true or a larger number — and that costs work proportional to the match count.
• Resource holding. Long-lived PITs keep segments pinned. Set a short keep_alive and renew it as you page.
• Scroll is a different tool. Scroll targets bulk export or offline processing of a large result set and holds a search context per scroll. For user-facing deep paging, PIT plus search_after is generally the lighter, more flexible choice.
• Index sorting is a lever with a price. index.sort.* aligns on-disk order with your query sort, which helps early termination, but it slows indexing and fixes the order at index creation.

How to check it actually works

1. While paging, write and delete documents between requests. Confirm the PIT walk neither duplicates nor skips, and compare against a non-PIT search_after walk to see the drift.
2. Assert every hit has a sort array of the expected length and that the tiebreaker value is present. Log the array you send back — a mangled value is the most common cause of a repeated page.
3. Measure heap and latency at increasing depth on your own cluster against your own SLOs. Do not import numbers from an article; shard count, document size, query shape and concurrency dominate.
4. Check the reference documentation for your exact version. PIT, _shard_doc, and the guidance favoring PIT over scroll have changed across 7.x, 8.x and later, and defaults are version-sensitive.

If you take one thing: stop treating deep pagination as a bigger window and start treating it as a cursor. The window is the symptom; the offset is the cause.