How can I use IDBKeyRange bounds with IDBCursor to paginate a Firefox IndexedDB object store without rescanning the entire dataset?
0 reputation · 26 Nov 2022, 12:49 UTC
0 reputation · 26 Nov 2022, 12:49 UTC
I want to page through a large dataset stored in an IndexedDB object store in Firefox, returning a fixed number of records per view. The dataset may be modified while the user navigates, so each page should reflect a consistent snapshot without rescanning the whole store for every request.
I plan to create an IDBKeyRange that defines the lower and upper bounds for the current page, open a cursor with that range, and then use cursor.continue to advance to the next page. However, I am unsure how to combine the bound with the cursor’s continuation to avoid restarting from the beginning, and what technique ensures stable pagination when records are inserted or deleted between pages.
How can I correctly set the key range for each page so that the cursor continues from where the previous page left off? What approach can I use to guarantee that each page shows a stable view of the data despite concurrent modifications?
26525 reputation · 26 Nov 2022, 13:04 UTC
Use a one‑sided IDBKeyRange.lowerBound that starts just after the last key you returned, open a forward‑only cursor (direction: 'next') inside a readonly transaction, collect the needed number of records, then remember the key of the last record. For the next page open a new cursor with IDBKeyRange.lowerBound(lastKey, false) (exclusive) to skip that key and continue from the next record.
IDBKeyRange.lowerBound(key, inclusive) positions the cursor at the first record whose key is ≥ key (if inclusive) or > key (if exclusive).IDBCursor.continue([key]) advances the iterator from its current position; supplying the last seen key resumes iteration exactly after that record.IDBKeyRange.lowerBound(0, true) for numeric keys). For subsequent pages, pass the exclusive lower bound: IDBKeyRange.lowerBound(lastKey, false).store.openCursor(range, 'next').lastKey.lastKey to build the new range.IndexedDB only guarantees transaction‑local snapshots. Because a transaction automatically commits when the micro‑task queue empties, you cannot keep a single transaction open across UI‑driven page requests. Consequently:
lastKey.Do you have a unique, monotonically increasing key (primary key or a unique index) that you can use as the pagination boundary? If the store only has non‑unique keys, the lower‑bound/exclusive technique may skip or duplicate records, and you would need to create a unique index or compound key before applying the method above.
// Example: fetch one page
function getPage(store, pageSize, lastKey) {
return new Promise((resolve, reject) => {
const tx = store.transaction(store.name, 'readonly');
const range = lastKey === undefined ? undefined : IDBKeyRange.lowerBound(lastKey, false);
const cursorReq = tx.objectStore(store.name).openCursor(range, 'next');
const page = [];
cursorReq.onsuccess = e => {
const cursor = e.target.result;
if (!cursor) { resolve(page); return; }
page.push(cursor.value);
if (page.length >= pageSize) {
// remember key for next page
lastKey = cursor.key;
resolve({ page, lastKey });
return;
}
cursor.continue();
};
cursorReq.onerror = e => reject(e.target.error);
});
}
Use comments to ask for clarification. Post a solution as an answer.
26,525 reputation · 26 Nov 2022, 18:38 UTC
When the object store uses a compound key (an array), IDBCursor.continue(key) works only if you pass the exact same key value. A more reliable approach is cursor.continuePrimaryKey(key, primaryKey) where key is the index key and primaryKey is the stored primary key of the last returned record. This tells the cursor to resume after that specific record, even if the index contains duplicate keys.
Open a new readonly transaction for every page. The transaction’s snapshot is taken when it starts, so the cursor will not see changes committed after the transaction began. Re‑using a single transaction across pages would hold locks and could block writes, and it would not reflect inserts that commit after the transaction started.
function fetchPage(store, pageSize, lastKey, lastPrimaryKey) {
return new Promise((resolve, reject) => {
const tx = store.transaction('readonly');
const idx = tx.index('someIndex'); // optional, use store if no index
let range;
if (lastKey === undefined) {
range = null; // start from beginning
} else {
// exclusive lower bound on the index key
range = IDBKeyRange.lowerBound(lastKey, false);
}
const cursorReq = idx.openCursor(range, 'next');
const results = [];
cursorReq.onsuccess = e => {
const cursor = e.target.result;
if (!cursor) { resolve(results); return; }
// If we stored a primaryKey, skip duplicates of the same index key
if (lastPrimaryKey !== undefined && cursor.primaryKey === lastPrimaryKey) {
cursor.continue(); // advance past the exact record
return;
}
results.push(cursor.value);
if (results.length === pageSize) {
// remember where we stopped
const stopKey = cursor.key;
const stopPrimaryKey = cursor.primaryKey;
cursor.continue(); // consume one more to position for next page
// we will return stopKey/stopPrimaryKey to caller
resolve({results, stopKey, stopPrimaryKey});
return;
}
cursor.continue();
};
cursorReq.onerror = e => reject(e.target.error);
});
}
This snippet shows how to pass both the index key and the primary key to avoid re‑returning the same record when the index is non‑unique, and how to obtain the bounds for the next request.