Diagnosing Terraform State Locking Failures with S3 & DynamoDB Backends

Problem: Terraform blocks with a state‑lock error

When you run terraform apply or terraform plan and see a message such as Error acquiring state lock: could not get lock on ..., Terraform is unable to obtain a lock on the remote state stored in an S3 bucket. The lock is held by a DynamoDB table that Terraform uses to serialize concurrent operations.

Takeaway

Most lock failures stem from one of five root causes:

• Missing or mis‑named DynamoDB table
• Incorrect IAM permissions for the Terraform process
• Stale lock items left by a previous run
• Wrong backend configuration (region, endpoint, etc.)
• Default 30‑second TTL too short for long operations

Diagnostic Flow

Use the table below to match the symptom you observe with the most likely cause, then follow the ordered checks and fixes.

Symptom	Likely Cause	Initial Check
Terraform hangs during lock acquisition, timeout after ~30s	Default TTL too short or network latency	Run TF_LOG=DEBUG terraform apply -input=false -auto-approve and look for Lock TTL logs.
Error “resource not found” for DynamoDB table	Missing or wrong table name	Check terraform show -json for backend config; verify table exists via AWS console or CLI.
Permission denied when Terraform accesses DynamoDB	IAM policy missing DynamoDB:PutItem, DeleteItem, GetItem	Run aws iam list-attached-user-policies --user-name and review permissions.
Lock item present with Owner not matching current run	Stale lock from interrupted run	Inspect lock item via aws dynamodb get-item and note TTL field.
Backend config uses wrong region or endpoint	Mis‑configured backend	Verify terraform init -backend-config=... output for region and endpoint details.

Ordered Checks & Fixes

1. Validate Backend Configuration

   Open backend.tf (or the file where the backend block resides). Confirm the bucket, key, region, and dynamodb_table values match your AWS resources. Example:

   terraform {
     backend "s3" {
       bucket         = "my-terraform-state"
       key            = "prod/terraform.tfstate"
       region         = "us-east-1"
       dynamodb_table = "terraform-state-lock"
     }
   }
   

   Run terraform init -backend-config=backend.tf and look for any errors pointing to a missing table or wrong region.

2. Check DynamoDB Table Existence

   Use the CLI:

   aws dynamodb describe-table --table-name terraform-state-lock
   

   If the table does not exist, create it:

   aws dynamodb create-table \
     --table-name terraform-state-lock \
     --attribute-definitions AttributeName=LockID,AttributeType=S \
     --key-schema AttributeName=LockID,KeyType=HASH \
     --billing-mode PAY_PER_REQUEST
   

   Ensure the table’s LockID matches the state file name (e.g., prod/terraform.tfstate).

3. Verify IAM Permissions

   Terraform needs the following actions on the DynamoDB table and S3 bucket:

   • DynamoDB: PutItem, DeleteItem, GetItem, UpdateItem
   • S3: s3:GetObject, s3:PutObject, s3:DeleteObject

   Attach a policy like:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "dynamodb:PutItem",
           "dynamodb:DeleteItem",
           "dynamodb:GetItem",
           "dynamodb:UpdateItem"
         ],
         "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/terraform-state-lock"
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:GetObject",
           "s3:PutObject",
           "s3:DeleteObject"
         ],
         "Resource": "arn:aws:s3:::my-terraform-state/*"
       }
     ]
   }
   

   Test the permissions by running terraform plan again.

4. Inspect and Clean Stale Lock Items

   List items in the lock table:

   aws dynamodb scan --table-name terraform-state-lock --query "Items[*].{LockID:LockID.S,TTL:TTL.N,Owner:Owner.S}" --output table
   

   If you see an entry with Owner not matching your current user and a TTL that has expired or is close to current epoch time, it is safe to delete:

   aws dynamodb delete-item \
     --table-name terraform-state-lock \
     --key '{"LockID":{"S":"prod/terraform.tfstate"}}'
   

   Only perform this deletion when you are certain no other Terraform process is running.

5. Adjust Lock TTL for Long‑Running Applies

   Terraform’s default lock TTL is 30 seconds. If your applies routinely exceed this, increase the TTL in the DynamoDB table by adding a ttl attribute or by using the lock_table_ttl backend option (Terraform 1.5+). Example:

   terraform {
     backend "s3" {
       dynamodb_table = "terraform-state-lock"
       lock_table_ttl = 300  # 5 minutes
     }
   }
   

   After changing the config, run terraform init -migrate-state to apply the new TTL.

6. Confirm Lock Release

   Execute a lightweight plan:

   terraform plan -input=false -no-color
   

   Check that no lock item remains in DynamoDB. Also, enable TF_LOG=DEBUG for a second run to see Lock acquired and Lock released messages.

Escalation Criteria

• If a lock persists after 5 minutes of repeated attempts, suspect network partitions or DynamoDB throttling. Check CloudWatch metrics for ConsumedReadCapacityUnits and ConsumedWriteCapacityUnits.
• If IAM policies are correct but lock acquisition still fails, involve your AWS security team to verify that no Service Control Policies or SCPs are blocking DynamoDB access.
• When you cannot resolve the lock issue within the team, open an incident with Ops or CloudOps, providing the TF_LOG output and DynamoDB table state.

Practical Checklist

• Confirm backend config matches resources.
• Verify DynamoDB table exists and is in the same region.
• Check IAM policy grants required DynamoDB and S3 actions.
• Inspect lock table for stale entries and delete if safe.
• Adjust lock TTL if applies regularly exceed 30s.
• Run a quick plan to ensure lock is released.

Limitations & Caveats

Manually deleting lock items can corrupt state if another process is running. Always coordinate with your team. Changing backend settings in production should be preceded by a state backup: terraform state pull > state.tfstate.backup. Granting overly broad IAM permissions can expose the lock store; follow the principle of least privilege.