Paging Large Deta Base Collections Without Timeouts or OOM

The problem: large scans fail inside Micro request budgets

When you need to export, re-index, or batch-process a Deta Base collection that has grown past a few thousand items, the obvious approach—issue one query and process everything—breaks against Deta Micro execution limits. Micros enforce request time and memory ceilings, and Deta Base does not expose an offset or skip parameter. The only supported way to walk a large collection is a forward cursor, so the engineering decision is not whether to paginate, but how to structure the pagination so a single invocation neither exhausts memory nor runs past the request deadline.

Decision constraints

Before choosing a paging strategy, fix the constraints that actually drive the design:

• No random access. Deta Base queries return a forward cursor in the response, not an offset. You cannot jump to an arbitrary page directly.
• Micro ceilings. Each invocation has a hard time budget and memory ceiling. A query that returns too many items or takes too long can fail mid-response.
• Concurrent writes. The cursor is not a transactional snapshot. Puts and deletes that happen while you scan can cause items to appear twice or be skipped entirely.
• Version drift. Exact query parameter names, maximum limit values, and default page sizes are version-sensitive. Verify them against your installed SDK before hard-coding constants.

Supported paging options compared

Option	Mechanism	Best for	Main risk
A. Cursor loop	Repeated query with a page limit and the last cursor until the cursor is empty	Exports, batch jobs, re-indexing	Strictly sequential; slow for very large sets
B. One-shot maximum query	Single query at the highest allowed limit	Small collections that fit comfortably in memory	Timeouts or out-of-memory errors as the collection grows
C. Fetch all, filter client-side	Pull everything, then filter in application code	Rarely appropriate for large sets	Defeats the purpose of server-side filtering
D. Parallel prefix scans	Split by key prefix or range and scan concurrently	Very large collections where sequential time is unacceptable	Requires key design that supports partitioning; more complex error handling

Option A is the default because it is the only approach that keeps memory bounded and requests independently retryable. Option D is the only path that parallelizes, but it depends on your key layout and adds coordination overhead. Options B and C are simple but do not scale.

Trade-offs that matter in practice

Page size is a budget decision. A moderate page size—typically a few hundred keys—keeps each request well under the Micro timeout. If a page fails, you retry only that page, not the entire scan. A larger page reduces round trips but increases the blast radius of a timeout.

Sequential means slow. Cursor paging cannot be parallelized within a single collection unless you partition by key. For a collection of millions of items, the wall-clock time of a sequential scan may exceed what a single Micro invocation allows, which is why you need a page cap and a resumable cursor.

At-least-once is the safe assumption. Because the cursor is not a snapshot, concurrent puts or deletes can make an item appear in two consecutive pages or vanish entirely. Design your handler to be idempotent—deduplicate by key and tolerate gaps—rather than assuming exactly-once delivery.

Concrete implementation: a resumable cursor loop

The following pattern keeps each request small, persists progress between invocations, and survives mid-scan failures. Adapt the exact method and field names to your installed SDK; the shape is consistent across the Python and Node SDKs.

# Run inside a Deta Micro or any environment with Base credentials.
# Verify the exact SDK signature for your version before relying on these names.

import os
from deta import Deta  # or the equivalent import for your SDK

# Meaningful placeholders: replace with your project key and Base name.
PROJECT_KEY = os.environ["DETA_PROJECT_KEY"]
BASE_NAME = "large-collection"
PAGE_SIZE = 200          # Moderate page size; adjust after timing tests.
MAX_PAGES = 50           # Hard cap per invocation to stay under the request budget.

db = Deta(PROJECT_KEY).Base(BASE_NAME)

def get_saved_cursor():
    # Replace with your persistence layer (another Base, a file, etc.).
    # Return None on first run.
    ...

def save_cursor(cursor):
    # Persist the cursor so a failed invocation resumes cheaply.
    ...

def process_batch(items):
    # Idempotent handler: dedupe by key and tolerate gaps.
    seen = set()
    for item in items:
        key = item["key"]
        if key in seen:
            continue
        seen.add(key)
        # ... your export or batch logic ...

cursor = get_saved_cursor()
pages = 0

while pages < MAX_PAGES:
    # The exact parameter names (limit, last) and response fields
    # (items, last) must match your installed SDK.
    res = db.query(
        limit=PAGE_SIZE,
        last=cursor
    )
    items = res.items
    if not items:
        # Empty page: scan complete. Clear the saved cursor.
        save_cursor(None)
        break

    process_batch(items)
    cursor = res.last
    if not cursor:
        # Cursor exhausted: scan complete.
        save_cursor(None)
        break

    save_cursor(cursor)
    pages += 1
else:
    # Page cap reached: exit cleanly so the next invocation resumes.
    # The saved cursor points to the next unprocessed page.
    pass

Where to run it: inside a Deta Micro HTTP handler, a scheduled action, or a local script with the project key exported as an environment variable. Required permissions: the project key must have read access to the target Base.

Expected checks: the loop terminates when the response cursor is empty or absent; the saved cursor advances monotonically; and the number of pages per invocation stays below the configured cap.

Relevant risks: if the SDK uses different field names than assumed, the loop will raise an attribute error on the first call. If the process is killed after processing a batch but before saving the cursor, the same page is re-processed—another reason the handler must be idempotent.

Validation: proving the scan is correct

A cursor loop that terminates is not necessarily a correct scan. Use these checks to validate behavior before trusting it in production:

1. Uniqueness and completeness. In a throwaway Base, insert a known number of records with distinct keys. Run the loop and assert that the collected key set equals the inserted set with no duplicates.
2. Termination. Confirm that the loop exits because the cursor became empty, not because it hit an exception or a page cap.
3. Concurrent-write behavior. Insert and delete records while the scan runs. Observe whether items appear twice or are skipped. This confirms the need for idempotent processing and tells you how much drift to expect under your write load.
4. Resumability. Persist the cursor, kill the process mid-scan, restart, and verify that the run resumes from the saved cursor without re-processing completed pages or skipping ahead.
5. Budget fit. Time a single maximal query versus the cursor loop inside a Micro. Record the latency and memory profile, then set the page size and page cap so the total stays comfortably inside the request ceiling.

Limitations and current verification

Deta Space hosted availability has changed over time, including a sunset announcement in 2024. Before building on this pattern, confirm that Deta Base and the SDK are still available for your account and region, and that your project is not affected by deprecation timelines.

Even where the service is available, treat the exact limit ceiling, default page size, and cursor semantics as version-sensitive. Do not hard-code a maximum limit without checking your installed SDK. The parameter and response field names shown above are illustrative; verify them against your environment before deploying.

Finally, remember that a Deta Base cursor is not a transactional snapshot. If your batch job cannot tolerate duplicates or gaps, you need an additional reconciliation step—such as comparing a collected count against an expected total your application maintains, or re-running the scan and diffing the results—rather than relying on the cursor alone.