Handling DynamoDB Pagination: Solving the 1 MB Result Limit

The 1 MB Pagination Problem

When executing a Query or Scan in DynamoDB, you cannot retrieve an entire dataset in a single API call if the result set exceeds 1 MB. Even if you set a high Limit value, DynamoDB will truncate the response once the total size of the items read reaches 1 MB. If more data remains, the service returns a LastEvaluatedKey.

The critical takeaway for engineers is that an empty Items list does not mean you have reached the end of the dataset. If a FilterExpression is used, DynamoDB reads up to 1 MB of data first and then filters it. If all items in that 1 MB block are filtered out, you receive an empty list but a valid LastEvaluatedKey. Stopping your loop at the first empty page results in data loss.

The Pagination Mechanism

Pagination in DynamoDB is a client-side responsibility. The process follows a specific request-response cycle:

• Initial Request: The client sends a Query or Scan request.
• Server Response: DynamoDB returns up to 1 MB of data and a LastEvaluatedKey if more items match the query.
• Subsequent Request: The client sends the same request but includes the LastEvaluatedKey as the ExclusiveStartKey.
• Termination: The loop continues until the response contains no LastEvaluatedKey.

Implementation Example (Node.js SDK v3)

The following example demonstrates a robust pagination loop. This must be run in an environment with AWS SDK v3 installed and credentials configured with dynamodb:Query permissions on the target table.

import { DynamoDBClient, QueryCommand } from "@aws-sdk/client-dynamodb";

const client = new DynamoDBClient({ region: "us-east-1" });

async function fetchAllItems(partitionKeyValue) {
  let items = [];
  let exclusiveStartKey = null;

  do {
    const params = {
      TableName: "YourTableName",
      KeyConditionExpression: "PK = :pk",
      ExpressionAttributeValues: { 
        ":pk": { S: partitionKeyValue } 
      },
      // Use ExclusiveStartKey to resume from the last page
      ExclusiveStartKey: exclusiveStartKey,
      // FilterExpression is applied AFTER the 1MB read
      FilterExpression: "attribute_exists(SomeAttribute)"
    };

    const command = new QueryCommand(params);
    const response = await client.send(command);

    if (response.Items) {
      items.push(...response.Items);
    }

    // Capture the key for the next iteration
    exclusiveStartKey = response.LastEvaluatedKey;

  } while (exclusiveStartKey);

  return items;
}

Diagnostic Check: Verifying Filter Behavior

To verify that your pagination logic handles filtered empty pages correctly, perform this test:

1. Create a dataset where the first 1 MB of items do not match your FilterExpression, but items beyond the 1 MB mark do.
2. Run your query. If your code stops when Items.length === 0, you will miss the remaining data.
3. Check the ConsumedCapacity by setting ReturnConsumedCapacity: "TOTAL". You will notice that you are charged for the full 1 MB read even if the Items array returned is empty.

Limits and Common Engineering Mistakes

The Limit Parameter Misconception

The Limit parameter does not define the maximum number of items returned to the client; it defines the maximum number of items evaluated before the operation stops. If you set Limit: 10 and use a FilterExpression, DynamoDB evaluates 10 items. If only 2 match the filter, you get 2 items back. If 0 match, you get 0 items back, but you may still receive a LastEvaluatedKey if there are more than 10 items in the table.

Performance and Cost Implications

Factor	Impact on Pagination
FilterExpression	Does not reduce Read Capacity Unit (RCU) cost. You pay for the 1 MB read regardless of how many items are filtered out.
Consistency	Strongly consistent reads consume double the RCU of eventually consistent reads per page.
Parallel Scans	Each segment maintains its own LastEvaluatedKey. You must track pagination state independently for every segment.

Fragility of the LastEvaluatedKey

The LastEvaluatedKey is an opaque token representing the last item processed. Avoid storing this key in a long-term database to resume a scan days later. If the table schema changes or the Global Secondary Index (GSI) is rebuilt, the key may become invalid or lead to inconsistent results.

Verification and Rollback

Verification: To confirm successful implementation, log the presence of LastEvaluatedKey and the size of the Items array for each loop iteration. Ensure the loop only terminates when LastEvaluatedKey is undefined.

Rollback: Since Query and Scan are read-only operations, there is no state change to roll back in the database. To revert client-side changes, return to the previous SDK version or remove the do...while loop logic.