Mango Query Pagination with Bookmarks: Handling Index Rebuilds

Question

Mango Pagination and Bookmarking
When using a Mango selector to retrieve a large collection, the limit and bookmark fields allow the client to iterate through results without using skip. The bookmark is a cursor that references a specific point in the index, enabling efficient forward paging.
Uncertain Behavior on Index Rebuild
In a dynamic database, indexes may be rebuilt automatically or manually. It is unclear whether a bookmark that was issued before an index rebuild remains valid afterward, and if not, how the client should detect and recover from that situation.
Concurrent Document Activity
While pagination is in progress, new documents may be inserted or existing ones deleted. The interaction between these changes and the bookmark’s position may affect the continuity of results, especially when the index is rebuilt in the middle of a paging session.
Open Questions

Does a bookmark issued against a specific index version survive an automatic or manual rebuild of that index?
If the bookmark becomes invalid, what response does the server return, and how can a client detect the need to restart pagination?
Are there recommended patterns for combining bookmarks with skip or startkey to mitigate issues caused by concurrent index changes?

Tasadduq Burney · Answer

Direct Answer
A Mango bookmark is an opaque cursor tied to the specific state of an index scan. When an index is rebuilt—whether manually via the _index endpoint or automatically by the database—the bookmark is invalidated. Because the underlying index structure is replaced, the previous position no longer exists. The server typically responds with an error (such as HTTP 400 or 500) or an empty result set, signaling the client to restart pagination from the beginning.

Index State vs. Document Activity
It is important to distinguish between a full index rebuild and concurrent document modifications:

Index Rebuilds: These are destructive to bookmarks. The cursor references a physical or logical location in a B-tree that is discarded during a rebuild.
Concurrent Activity: Insertions or deletions do not invalidate the bookmark itself. However, because bookmarks track index position rather than a point-in-time snapshot, documents inserted before the current cursor position will not appear in subsequent pages, while those inserted after will. This may lead to minor skipping or duplication across page boundaries.

Detecting and Recovering from Invalid Bookmarks
Clients should implement a recovery loop to handle stale bookmarks. Monitor for the following responses:

HTTP 400 Bad Request: Often accompanied by an "invalid bookmark" error message.
HTTP 500 Internal Server Error: Occurs if the server cannot reconcile the cursor with the current index.
Empty docs array: If total_rows is greater than zero but the docs array is empty, the bookmark may have pointed to a position that no longer exists.

Upon detecting these, the client should discard the current bookmark and re-issue the original query starting from the first page.

Recommended Implementation Patterns

Exclusive Bookmark Use: Do not combine bookmark with skip. Using skip forces the server to scan through discarded records, negating the performance benefits of the bookmark and creating unpredictable offsets.
Selector Persistence: Store the original Mango selector on the client side so the query can be re-issued identically after a restart.
Error-Triggered Reset: Implement a try-catch block around the pagination request that triggers a full reset when a 400/500 error is encountered during a bookmarked request.

Verification Steps
To verify this behavior in a development environment (e.g., CouchDB 3.x), execute the following sequence:
# 1. Fetch the first page to obtain a bookmark
curl -X POST http://localhost:5984/db/_find -d '{"selector":{"type":"test"},"limit":10}'

# 2. Manually trigger an index rebuild
curl -X POST http://localhost:5984/db/_index -d '{"ddoc":"test-design","name":"test-index","type":"json","def":{"fields":[{"type":"asc"}]}}'

# 3. Attempt to use the bookmark from step 1
curl -X POST http://localhost:5984/db/_find -d '{"selector":{"type":"test"},"limit":10,"bookmark":"OLD_BOOKMARK_VALUE"}'

Assumptions and Uncertainty

This behavior is based on CouchDB 3.x and Cloudant implementations; other Mango-compatible engines may handle cursor invalidation differently.
Exact HTTP status codes are not strictly standardized across all versions; clients should treat any unexpected error during a bookmarked request as a potential need for a reset.

If you encounter a specific error code not listed here, please provide the response body for further analysis.

Mango Query Pagination with Bookmarks: Handling Index Rebuilds

Mango Pagination and Bookmarking

Uncertain Behavior on Index Rebuild

Concurrent Document Activity

Open Questions

1 answer

Direct Answer

Index State vs. Document Activity

Detecting and Recovering from Invalid Bookmarks

Recommended Implementation Patterns

Verification Steps

Assumptions and Uncertainty

Your perspective belongs here.

0 question comments