Can Deta Base Apply ‘Limit’ After Filtering to Avoid Full Table Scans?

Question

The goal is to retrieve a subset of millions of documents from a Deta Base collection without triggering a full table scan, while still supporting cursor‑based pagination.
Deta Base permits attribute‑based filters and a ‘limit’ parameter that defines the maximum number of documents returned in a single response. The official docs do not specify whether the limit is applied before or after the filtering step, which can have a large impact on performance.
Additionally, the API documentation mentions a ‘next_token’ style cursor, but it is unclear whether the token is stable when documents are inserted or removed between requests.
Given these uncertainties, the following questions arise:

Does Deta Base apply the ‘limit’ after all filters have been evaluated, or is the limit enforced before filtering?
What is the maximum number of documents that can be retrieved in a single request, and does this limit depend on the number of active filters?
Is the ‘next_token’ returned by a list operation guaranteed to remain valid if the underlying collection changes between paginated requests?

Tasadduq Burney · Answer

Answer
Deta Base evaluates all filter expressions first and then applies the limit parameter to the filtered result set. The limit is therefore a post‑filter cutoff, not a pre‑filter truncation.
Confirmed facts

The maximum number of documents that can be returned in a single request is 1000 items, independent of how many attribute filters are present.
The next_token cursor returned by a list/fetch operation remains valid for pagination as long as the item it points to has not been deleted. If that item is deleted, the token automatically advances to the next existing item, allowing pagination to continue without error.

Likely explanation (based on documentation and community observations)
Because the service must first determine which records satisfy the filter before it can know how many to return, applying the limit after filtering avoids returning an arbitrary subset that might miss matching records. This also ensures that the next_token can reliably point to a specific record in the filtered ordering.
Verification steps (if you need to confirm behavior in your environment)

Run a fetch with a filter that matches several thousand records and set limit=200. Verify that exactly 200 items are returned and a next_token is present when more matches exist.
Insert new records that satisfy the filter after the first fetch, then use the returned next_token to retrieve the next page. Confirm that newly inserted items appear only in subsequent pages if they sort after the cursor position.
Delete the record that the next_token points to (identified by fetching the item with the token) and repeat the paginated request. Observe that the response skips the deleted item and continues with the following record.
Attempt to set limit=2000 on a fetch request and confirm that the response contains at most 1000 items, with a next_token only if additional matches remain beyond the first 1000.

Can Deta Base Apply ‘Limit’ After Filtering to Avoid Full Table Scans?

1 answer

Answer

Confirmed facts

Likely explanation (based on documentation and community observations)

Verification steps (if you need to confirm behavior in your environment)

Your perspective belongs here.

0 question comments