Resolving Pulumi State Lock Conflicts in Shared Backends

The State Lock Problem

When running pulumi up or pulumi destroy, you may encounter a "Conflict" or "Lock" error. This happens because Pulumi implements a locking mechanism to prevent concurrent updates to the same stack, which would otherwise lead to state corruption or race conditions in your cloud infrastructure.

The immediate takeaway: Do not force-unlock a stack unless you have verified that no other process is actively modifying your infrastructure. Unlocking a live deployment can corrupt your state file, requiring manual recovery from backups.

Diagnostic Matrix

Use this table to identify the likely cause of the lock based on your current environment.

Symptom	Likely Cause	Risk Level
Error appears immediately in CI/CD	Overlapping pipeline triggers or a hanging runner	Low (if process is dead)
Error appears during manual CLI use	Another teammate is deploying to the same stack	High (if unlocked prematurely)
Error persists after process termination	Interrupted network connection during state write	Medium (state may be stale)

Step-by-Step Resolution

Step 1: Verify Active Deployments

Before attempting to clear a lock, confirm that no legitimate process is running. Check the following:

• CI/CD Dashboards: Check GitHub Actions, GitLab CI, or Jenkins for any "Running" or "Pending" jobs targeting the specific stack.
• Pulumi Console: If using the Pulumi Service backend, check the "Activity" tab for the stack to see if an update is currently marked as in-progress.
• Team Communication: Confirm with other engineers if they are performing a manual update from their local terminal.

Step 2: Terminate Orphaned Processes

If a local process crashed or a CI runner timed out, the lock may remain even if the process is no longer active. If you have access to the machine that initiated the update, ensure the Pulumi CLI process is fully terminated.

Step 3: Manually Release the Lock

If you have confirmed that no active deployment is occurring, you can manually release the lock using the CLI. This command tells the backend to ignore the existing lock and allow new operations.

# Run this from your local terminal or CI runner
# Required Permissions: Stack administrator or equivalent backend write access
pulumi stack unlock [stack-name]

Risk: If you run this while a process is actually updating resources, you risk a "split-brain" scenario where two processes attempt to write to the state file simultaneously.

Verification and Recovery

After unlocking, you must verify that the state is consistent before proceeding with new changes.

1. Export State: Run pulumi stack export > state_backup.json. This ensures the state is readable and provides a manual backup before you attempt a refresh.
2. Synchronize State: Run pulumi refresh. This compares the state file against the actual resources in your cloud provider. If the previous interrupted process partially completed its work, refresh will bring the state file up to date without attempting to modify infrastructure.
3. Test Update: Perform a low-risk change, such as updating a resource tag, and run pulumi up to confirm the lock is fully cleared and the pipeline is functional.

Escalation Criteria

If pulumi stack unlock fails or the lock immediately reappears, follow these escalation steps:

• Backend Permissions: Verify that your current API token or cloud credentials have the necessary permissions to modify the state file in S3, GCS, or the Pulumi Service.
• Backend-Specific Cleanup: For self-managed backends (S3/GCS), check for orphaned .lock files in the bucket. Caution: Only delete these as a last resort after backing up the entire bucket.
• Pulumi Support: If using the Pulumi Service and the lock persists despite no active processes, contact Pulumi support to clear the server-side lock.