DynamoDB Scan and FilterExpression Pagination Boundary

Answer

Do not rely on Scan + FilterExpression to guarantee a minimum number of filtered items per request. Move filtering to the key when possible. If post-read filtering is unavoidable, accumulate filtered items client-side with a bounded evaluation budget and always continue pagination when LastEvaluatedKey is present, even if Items is empty.

Confirmed facts

• FilterExpression is applied after items are read from storage and before results are returned.
• Limit and the 1 MB response size limit apply to evaluated items, not to items that pass the filter.
• DynamoDB can return an empty Items list with a LastEvaluatedKey. That means the evaluation boundary was reached with zero matches in that segment.
• RCU is charged for evaluated items, not returned items. Empty filtered pages still consume capacity.
• LastEvaluatedKey presence is the only continuation signal. It does not indicate whether future pages will contain matches.

Likely explanation for empty pages and RCU waste

With low-selectivity filters on Scan, many evaluated items are discarded. Limit stops evaluation after N items are read, regardless of how many survive the filter. Deep pagination repeatedly evaluates non-matching items, which is expected behavior for Scan with a post-read filter.

Recommended pattern

1. Prefer key-level filtering. Use Query with KeyConditionExpression on the partition and sort key, and add a Global Secondary Index where the filterable attribute is part of the key. This reduces evaluated items and eliminates empty filtered pages.
2. If FilterExpression must stay, accumulate client-side:
   • Pick a fixed evaluation Limit per call, e.g., Limit = 100 or a size that keeps response under 1 MB.
   • Loop: Scan with FilterExpression, same Limit, ExclusiveStartKey = previous LastEvaluatedKey.
   • Append items that pass the filter to a local buffer.
   • Stop when buffer reaches the desired page size or LastEvaluatedKey is null.
   • Cap total evaluations per logical request using ConsumedCapacity to bound RCU.
3. Set a pagination depth limit for Scan workloads and consider export or batch processing for full table scans.

Handling LastEvaluatedKey with zero items

Treat it as continuation, not end of data.

• Do not return empty to the caller.
• Use the returned LastEvaluatedKey as ExclusiveStartKey for the next Scan with the same FilterExpression and Limit.
• Continue accumulation until the desired filtered count is met or LastEvaluatedKey is null.
• Monitor ConsumedCapacity per call to detect runaway RCU on deep scans.

Verification steps: test with mixed matching and non-matching items, observe empty Items with LastEvaluatedKey, and confirm ConsumedCapacity reflects evaluated items before filtering.