Diagnosing and Fixing Pulumi State Lock Failures During Stack Updates

Recognizable Condition

When a pulumi up, pulumi refresh, or any stack‑modifying command stalls or fails with messages such as Failed to acquire state lock or Another operation is in progress, the Pulumi CLI is unable to obtain the exclusive lock that protects the stack’s state file. This lock is required to prevent concurrent modifications that could corrupt the state.

Cause & Diagnostic Table

Possible Cause	Typical Symptoms	Diagnostic Check
Concurrent updates from multiple CLI or automation jobs	Command hangs for minutes; error shows lock acquisition failure	Check running processes and CI/CD job logs for the same stack name
Stale lock entry left by a crashed or interrupted run	Lock error persists even after all jobs appear stopped	Inspect the lock backend (e.g., DynamoDB table) for an existing lock item
Missing or unhealthy lock backend (e.g., DynamoDB table not created)	Error mentions missing backend resource or access denied	Verify backend resource exists and is in an ACTIVE state
Insufficient IAM permissions for lock operations	Permission denied errors when Pulumi tries to GetItem/PutItem/DeleteItem	Review the IAM policy attached to the role or user used by Pulumi
Interrupted run that left the state file partially updated	Subsequent updates fail or state shows inconsistencies	Check recent update logs for abrupt exits or network failures

Ordered Checks

1. Confirm No Parallel Jobs
   Run ps -ef | grep pulumi on the machine, and review CI/CD pipelines for any jobs that might still be running against the same stack. If any are found, wait for them to finish or cancel them.
2. Validate Lock Backend Health
   For an S3 backend with DynamoDB locking, use:

   aws dynamodb describe-table --table-name pulumi-lock-
   

   Verify the table status is ACTIVE and that the endpoint is reachable.
3. Check IAM Permissions
   Ensure the user/role executing Pulumi has at least:

   dynamodb:GetItem
   

   dynamodb:PutItem
   

   dynamodb:DeleteItem
   

   If using a custom policy, add these actions for the lock table ARN.
4. Inspect Recent Logs for Abrupt Termination
   Run pulumi stack history --stack and look for entries with status FAILED or ABORTED. Correlate timestamps with your system’s process logs.
5. Attempt Manual Lock Release (if safe)
   If you have confirmed no other updates are running, you can manually release the lock:

   pulumi cancel --yes
   

   This command removes the lock entry from the backend. Use only when you are certain the stack is idle.

Fixes Tied to Findings

• Concurrent Updates – Serialize CI/CD triggers or add a pre‑step that checks for an existing lock using pulumi stack --show-outputs before starting the deployment.
• Stale Lock Entry – After confirming no active run, delete the lock item directly:

  aws dynamodb delete-item \
    --table-name pulumi-lock- \
    --key '{"StackName":{"S":""}}'
  

• Missing Lock Backend – Create the DynamoDB table with the schema Pulumi expects:

  aws dynamodb create-table \
    --table-name pulumi-lock- \
    --attribute-definitions AttributeName=StackName,AttributeType=S \
    --key-schema AttributeName=StackName,KeyType=HASH \
    --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1
  

• Insufficient IAM Permissions – Attach the following policy snippet to the role/user:

  {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["dynamodb:GetItem","dynamodb:PutItem","dynamodb:DeleteItem"],"Resource":"arn:aws:dynamodb:*:*:table/pulumi-lock-"}]}
  

• Interrupted Run – Re‑run the update after a clean lock release, and consider adding a pulumi refresh step to reconcile state.

Escalation Criteria

• Lock persists after manual release attempts.
• Backend health checks (e.g., DynamoDB describe‑table) repeatedly fail.
• Multiple stacks in the same environment exhibit lock failures, suggesting a systemic configuration or permission issue.
• You lack the necessary permissions to modify the lock backend or IAM policies.

Verification Steps

1. Run pulumi stack select <stack-name> followed by pulumi up --refresh. Observe that the lock is acquired within a few seconds and released without error.
2. Check the backend (e.g., DynamoDB) to confirm GetItem, PutItem, and DeleteItem operations succeeded during the run by enabling CloudWatch metrics for the table.
3. Verify pulumi stack history shows a successful update and that no subsequent runs produce lock errors.

Limitations and Precautions

• Deleting a lock entry while an update is in progress can corrupt the state. Always ensure the stack is idle before manual intervention.
• Modifying lock backend configuration should be tested in a non‑production stack first to avoid unintended state migration issues.
• Pulumi’s lock mechanism is backend‑specific; the commands above assume an S3 backend with DynamoDB locking. For other backends (e.g., Azure Blob Storage), adjust the lock resource accordingly.

Practical Example: S3 Backend with DynamoDB Lock

Assume a stack named dev-app configured with the following Pulumi configuration:

pulumi config set aws:region us-west-2
pulumi config set pulumi:backend s3://my-pulumi-state/dev-app
# DynamoDB lock table is created automatically by Pulumi if not present

During a pipeline run, the command fails with Failed to acquire state lock. Following the diagnostic guide:

1. Check for other runs: ps -ef | grep pulumi shows none.
2. Describe the lock table: aws dynamodb describe-table --table-name pulumi-lock-dev-app returns status ACTIVE.
3. IAM policy review confirms dynamodb:GetItem missing; add it.
4. Re‑run pulumi up – lock is acquired successfully.

After the fix, pulumi stack history shows a successful update, and subsequent runs complete without lock errors.