Using SourceEngine Scroll API for Efficient Deep Pagination

Quick answer

The SourceEngine Scroll API lets you retrieve large result sets in small, manageable batches without the performance penalty of deep offset‑based pagination. It works by creating a point‑in‑time snapshot of the index and returning a scroll_id cursor that you reuse to fetch the next batch until the dataset is exhausted.

How the Scroll API works – a worked example

Below is a minimal, reproducible sequence using curl against a local SourceEngine node running on localhost:9200. Adjust the index name, query, and timeout as needed for your environment.

1. Start the scroll – send a search request that includes the scroll parameter to keep the context alive. The size controls how many hits are returned per batch.

curl -X POST "localhost:9200/my-index/_search?scroll=1m" -H 'Content-Type: application/json' -d '
{
  "size": 1000,
  "query": {
    "match_all": {}
  }
}'

The response contains two important fields:

• _scroll_id – the cursor you will use for subsequent requests.
• hits.hits – the first batch of documents (up to the size you specified).

1. Fetch the next batch – POST the scroll ID to the /_search/scroll endpoint, keeping the same scroll timeout (or a shorter one if you prefer).

curl -X POST "localhost:9200/_search/scroll" -H 'Content-Type: application/json' -d '
{
  "scroll": "1m",
  "scroll_id": "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4W..."
}'

Repeat step 2 until the returned hits.hits array is empty, indicating that all matching documents have been streamed.

1. Close the scroll context – explicitly delete the cursor to free server resources.

curl -X DELETE "localhost:9200/_scroll" -H 'Content-Type: application/json' -d '
{
  "scroll_id": ["DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4W...", "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD5Z..."]
}
'

If you used only one scroll ID, the array can contain a single element.

Limits and practical considerations

Batch size and memory

The default batch size is 10 000 hits. Increasing size reduces the number of round‑trips but raises memory consumption on both the SourceEngine node (to hold the snapshot) and the client (to accumulate the response). Very large batches (e.g., > 50 000 hits) can trigger out‑of‑memory errors, especially under concurrent scrolls.

Scroll timeout

The scroll parameter (e.g., 1m) defines how long the context remains valid between requests. If you exceed this interval before issuing the next /_search/scroll call, the cursor expires and you must restart the scroll from the beginning. Choose a timeout that comfortably exceeds your expected processing time per batch.

Read‑only snapshot

A scroll provides a point‑in‑time view of the index as it existed when the initial search was executed. Documents added, updated, or deleted after that point are not reflected in the scroll results. This guarantees a stable ordering but can lead to stale data if the index changes rapidly and you need up‑to‑date information.

Resource cleanup

Each scroll context consumes a small amount of heap and a file descriptor on the cluster. Forgetting to call the delete/_scroll API leaves the context alive until the timeout expires, which can exhaust the scroll pool and block other queries. Always close the cursor in a finally block or equivalent error‑handling construct.

Common mistakes to avoid

• Using scroll for real‑time user‑facing pagination – because the snapshot is static, users may see outdated results if the underlying data changes while they page through results. For interactive UI, prefer search_after or a point‑in‑time (PIT) API if your version supports it.
• Setting an excessively large size – monitor heap usage on both client and node; increase only after verifying that memory stays within safe limits.
• Neglecting to close the scroll – automate cleanup; a missing delete call is a frequent source of resource leaks in long‑running batch jobs.
• Assuming ordering is immutable across updates – while the scroll guarantees a consistent view of the index at the start, the internal sort order may differ from a fresh query if the index mapping or analyzer changes.

Verification steps (perform in a test environment)

1. Run the initial search request with a modest size (e.g., 100) and a short scroll timeout (e.g., 30s). Record the returned _scroll_id.
2. Repeatedly call /_search/scroll with that ID, confirming each response returns a new batch of hits until the batch is empty.
3. While scrolling, monitor the node’s JVM heap (via /_nodes/stats/jvm) to ensure it stays within expected bounds for your chosen batch size.
4. After the loop, issue the delete/_scroll request and verify that /_cat/scrolls?v shows zero active scroll contexts (or that the count decreases by the number you closed).

These steps help you observe the behavior described above; they are not a guarantee of production‑ready performance and should be adapted to your specific workload and version of SourceEngine.