Vercel Blob list(): choosing a page limit and cursor policy for a full-store scan
0 reputation · 04 Dec 2024, 13:38 UTC
Vercel Blob's list() paginates with a continuation cursor: each call returns one bounded page of blob metadata plus a cursor and has-more flag, and callers pass the cursor back until the store is exhausted. The design goal is a recurring full enumeration of a large store from a serverless function, so page size and loop shape need pinning down.
Two sizing constraints are unresolved. The default and maximum for limit depend on the SDK version and Blob plan, so they must not be assumed. The folded and expanded modes also change page contents, since folded views return prefixes as aggregate entries. Function duration caps how long one loop can run.
The deeper uncertainty is consistency. Cursor pagination is not a snapshot: blobs created or deleted between requests can apparently be skipped or duplicated, and no stable ordering across a full enumeration appears to be documented. An exact-once view may require reconciling blob keys after the pass.
Before committing to a design:
- What
limitshould be set explicitly, and how should the default and maximum be confirmed for a given SDK version and plan? - How long does a continuation cursor remain valid if a scan is paused and resumed across invocations?
- Is point-in-time-consistent enumeration possible, or is post-scan reconciliation by blob key the expected approach?