Elasticsearch Search‑After: Fast Deep Pagination Without From/Size Overhead

Why Deep Pagination Slows Down

When you ask Elasticsearch to return, say, the 10,000th page of a 1‑million‑document index with from/size, the engine still has to scan the first 999,900 documents to reach that point. That linear scan turns a simple 10‑millisecond request into a sluggish operation that can hit the server’s memory limits and increase query latency dramatically.

Enter Search‑After: A Cursor‑Based Alternative

Search‑After is a stateless, cursor‑based pagination method built into Elasticsearch’s Query DSL. Instead of telling the engine how many documents to skip, you hand it the sort values of the last document you received. Elasticsearch then continues the search from that exact point, without re‑scanning the earlier part of the result set.

Core Mechanics

• Sort requirement: You must specify a sort clause. The sort must be deterministic and, ideally, unique (e.g., timestamp + _id).
• Cursor capture: After each query, take the sort array from the last hit and store it.
• Next request: Pass that array to the search_after parameter in the next query.

Implementation Example

# 1. Create a test index with a composite sort field
PUT /my_index
{
  "mappings": {
    "properties": {
      "@timestamp": {"type": "date"},
      "msg": {"type": "text"}
    }
  }
}

# 2. Bulk‑index 100k documents (pseudo‑code, run in Kibana Dev Tools)
POST /my_index/_bulk
{ "index": {"_id": "1"} }
{ "@timestamp": "2026-10-01T00:00:00Z", "msg": "first" }
…

# 3. First page: fetch 100 hits sorted by timestamp and _id
GET /my_index/_search
{
  "size": 100,
  "sort": [
    {"@timestamp": "asc"},
    {"_id": "asc"}
  ],
  "query": {"match_all": {}}
}

# 4. Capture the last hit’s sort values
# Suppose the response contains:
# "hits": { "hits": [ …, { "sort": ["2026-10-01T00:00:00Z", "100"] } ] }
# Store: ["2026-10-01T00:00:00Z", "100"]

# 5. Next page using search_after
GET /my_index/_search
{
  "size": 100,
  "sort": [
    {"@timestamp": "asc"},
    {"_id": "asc"}
  ],
  "search_after": ["2026-10-01T00:00:00Z", "100"],
  "query": {"match_all": {}}
}

Run each request on the same cluster node, ensuring you have cluster:monitor permissions if you’re using the Dev Tools console. After each page, verify that the hits.total remains consistent across the two methods (from/size vs. search_after).

Performance Verification

1. Measure the latency of page 1, 10, 50, and 100 using from/size. You’ll see latency climb linearly.
2. Repeat the same sequence with search_after. Latency should stay flat after the first page.
3. Compute a checksum of the sorted _id values returned by both methods and confirm they match.

Trade‑offs and Limitations

• No random access: You can’t jump directly to page 500. The client must maintain the last cursor value.
• Stable sort key required: If documents are inserted or updated with a timestamp that falls between two pages, you may see duplicates or gaps.
• Unique sort necessary: Without a unique tie‑breaker (like _id), the order can become nondeterministic, leading to overlapping results.
• Index updates during pagination: Concurrent writes can cause missed or repeated documents. For real‑time dashboards, consider snapshotting the index or using a read‑only replica.

When to Use Search‑After

• Large result sets where deep paging is required (e.g., analytics dashboards, export tools).
• Real‑time or near‑real‑time search where from/size would be too expensive.
• Environments where you can guarantee a stable, unique sort order.

Actionable Checklist

1. Define a deterministic, unique sort (timestamp + _id).
2. Implement the cursor capture logic in your client (store search_after array).
3. Replace any deep from/size pagination with search_after calls.
4. Run a simple latency test to confirm linear performance.
5. Document the requirement for stable sort keys in your API spec.