Diagnosing and Fixing DynamoDB Throughput & Hot‑Partition Issues in Provisioned & On‑Demand Modes

Recognizable Conditions

When you see sustained high latency, frequent ProvisionedThroughputExceededException errors, or a spike in ThrottledRequests CloudWatch metric, suspect one of the following:

• Overall table capacity is lower than the workload demands.
• Traffic is concentrated on a single partition key value (hot partition).
• Item sizes are close to the 400 KB limit, consuming more capacity units than expected.
• Auto‑scaling is too conservative or misconfigured.
• Read/write ratio is imbalanced, masking capacity issues.

Cause/Diagnostic Table

Observed Symptom	Likely Cause	Primary Metric
High latency (>200 ms)	Throttling due to insufficient capacity	ConsumedReadCapacityUnits / ConsumedWriteCapacityUnits
ProvisionedThroughputExceededException	Hot partition or capacity mis‑scaling	ThrottledRequests, PartitionKeyDistribution
Consistent reads > 1 GB/s	Item size close to 400 KB limit	ItemSizeDistribution
Auto‑scaling not reacting to traffic spikes	Target utilization too low or scaling steps too small	AutoScalingTargetUtilization, ScalingStepSize
90% reads, 10% writes but writes throttled	Imbalanced read/write ratio	Read/Write Capacity Units

Ordered Checks

1. Verify Current Capacity

   aws dynamodb describe-table \
     --table-name <TABLE_NAME> \
     --query 'Table.{ProvisionedThroughput:ProvisionedThroughput, BillingModeSummary:BillingModeSummary}'
   

   Check ProvisionedThroughput (read/write units) and BillingModeSummary.BillingMode (PROVISIONED or PAY_PER_REQUEST). For On‑Demand tables, confirm that the ProvisionedThroughput fields are omitted.

2. Inspect CloudWatch Metrics

   aws cloudwatch get-metric-statistics \
     --namespace AWS/DynamoDB \
     --metric-name ThrottledRequests \
     --dimensions Name=TableName,Value=<TABLE_NAME> \
     --statistics Sum \
     --period 300 \
     --start-time $(date -u -d '10 minutes ago' +%FT%TZ) \
     --end-time $(date -u +%FT%TZ)
   

   Look for sustained Sum > 0. If present, throttling is occurring.

3. Analyze Partition Key Distribution

   aws dynamodb scan \
     --table-name <TABLE_NAME> \
     --limit 1000 \
     --select 'COUNT' \
     --projection-expression <PARTITION_KEY> \
     --output json | jq '.Items | group_by(.<PARTITION_KEY>.S) | map({key: .[0].<PARTITION_KEY>.S, count: length})'
   

   Identify if a single key value accounts for >30% of requests. That indicates a hot partition.

4. Measure Item Size

   aws dynamodb scan \
     --table-name <TABLE_NAME> \
     --limit 100 \
     --output json | jq \
       '.Items | map(map_values(.S? | length)) | add'
   

   Average size close to 400 KB suggests higher capacity consumption per request.

5. Check Auto‑Scaling Policy (if applicable)

   aws dynamodb describe-table \
     --table-name <TABLE_NAME> \
     --query 'Table.AutoScalingSettings'
   

   Verify TargetValue (default 70%) and ScaleInCooldown/ScaleOutCooldown. A target < 50% may delay scaling.

Fixes Tied to Findings

• Insufficient Overall Capacity
  • Provisioned mode: increase ReadCapacityUnits and/or WriteCapacityUnits via aws dynamodb update-table or enable auto‑scaling.
  • On‑Demand: confirm that traffic is within the per‑region limits (default 40 GB/s). If exceeded, request a limit increase.
• Hot Partition Detected
  • Redesign partition key to spread traffic (e.g., add a hash or time component).
  • Use GlobalSecondaryIndex with a different partition key if the hot key is a query filter.
  • Consider BatchWriteItem with random prefixes to balance writes.
• Large Item Sizes
  • Normalize data: move infrequently accessed attributes to a separate table or a document store.
  • Use StreamProcessor to materialize large items into smaller chunks.
• Auto‑Scaling Mis‑Configured
  • Adjust TargetValue to 80–90% for more aggressive scaling.
  • Increase ScaleOutCooldown to allow new capacity to be provisioned before further scaling.
  • Set ScaleInCooldown to 300 s to avoid rapid down‑scaling during transient spikes.
• Read/Write Imbalance
  • Provision separate read and write capacity units if you have a distinct read/write ratio.
  • Enable ReadCapacityUnits auto‑scaling independently of WriteCapacityUnits to keep reads fast.

Escalation Criteria

1. If throttling persists after capacity adjustments and auto‑scaling is correctly configured.
2. When hot‑partition analysis reveals that redesigning the key is not feasible within the current application architecture.
3. If CloudWatch metrics show that ConsumedCapacityUnits exceed 80% of provisioned capacity for >30 minutes despite scaling.
4. When cost spikes exceed budget thresholds after scaling changes.

At this point, open an AWS Support case with details of the metrics and the steps already taken. Include the TableName, BillingMode, and any auto‑scaling policy JSON.

Practical Verification Checklist

• Run aws dynamodb describe-table to confirm new capacity values.
• Re‑run CloudWatch queries to ensure ThrottledRequests sum is now 0.
• Perform a load test (e.g., using aws dynamodb batch-get-item) and monitor ConsumedReadCapacityUnits in real time.
• Check the AutoScalingPolicy JSON for correct target and cooldown values.

Remember that every capacity increase or policy change increases operational cost. Monitor EstimatedCharges in the AWS Billing console to stay within budget.