Moving a Pulumi Project to an S3 Self-Managed Backend (With Locking Caveats)

The problem and the payoff

By default, Pulumi stores stack state on the Pulumi Cloud service or in a local file. If your organization wants state to live in your own AWS account — for data residency, cost, or access-control reasons — the self-managed backend lets you point Pulumi at an S3 bucket you control. The payoff is shared, durable state for a team without handing it to a third party. The catch: you become responsible for durability, access control, and update serialization, and locking behavior on self-managed backends is version- and backend-specific, so you must verify it before trusting concurrent runs.

Desired outcome

After this guide, pulumi preview and pulumi up for your project read and write state in s3://<bucket>/<prefix> using your AWS credentials, state is versioned and encrypted at rest, and you have a tested recovery path that does not involve editing state JSON by hand.

Prerequisites

• Pulumi CLI installed; check with pulumi version. Behavior described here assumes a current 3.x CLI — verify against the docs for your exact version.
• AWS credentials with access to the state bucket (environment variables, shared config, or an IAM role in CI).
• An existing Pulumi project you want to migrate, or a new one.
• Permission to create an S3 bucket and, if your CLI/backend version supports it, a DynamoDB table for state locking.

Step 1: Create the state bucket with baseline controls

Create a dedicated bucket — do not share it with application data. Enable versioning (your main recovery mechanism) and default encryption (SSE-S3 is fine; SSE-KMS if you need key-level audit). Block all public access. Example with the AWS CLI, run from any machine with admin-level AWS credentials:

aws s3api create-bucket --bucket my-org-pulumi-state --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1
aws s3api put-bucket-versioning --bucket my-org-pulumi-state \
  --versioning-configuration Status=Enabled
aws s3api put-bucket-encryption --bucket my-org-pulumi-state \
  --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

Expected check: aws s3api get-bucket-versioning --bucket my-org-pulumi-state returns Enabled. Risk: bucket names are globally unique; pick a convention like <org>-pulumi-state-<region> and document it so teammates and CI use the same bucket, prefix, and region — inconsistency here causes split state.

Step 2: Scope IAM to least privilege

Create a policy granting s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:ListBucket on the state bucket and its objects only, plus s3:GetObjectVersion if you want version-based recovery. Attach it to the user or CI role that runs Pulumi. Broad s3:* on all buckets increases blast radius if credentials leak. If you enable DynamoDB locking (next step), add dynamodb:GetItem, PutItem, and DeleteItem on the lock table only.

Step 3: Verify locking support before relying on it

This is the step most guides skip. Whether the S3 backend supports DynamoDB-based locking — and how it is enabled — has changed across CLI versions. Before promising mutual exclusion, check the Pulumi documentation for your CLI version (pulumi version) for the S3 backend's locking options. If locking is unavailable or unverified, treat concurrent pulumi up runs as unsafe and serialize updates in CI (a pipeline-level mutex or single-concurrency setting) until you have confirmed otherwise. Concurrent updates without locking can corrupt state.

Step 4: Log in to the backend and migrate the stack

Run these from the project directory, with the AWS credentials from step 2 active:

pulumi login s3://my-org-pulumi-state/my-project
pulumi stack export --stack dev --file dev-state-backup.json   # if migrating an existing stack
pulumi stack init dev   # or: pulumi stack select dev after importing

For an existing stack on another backend, export first (as above), log in to the S3 backend, create the stack, then pulumi stack import --file dev-state-backup.json. Keep the export file somewhere safe and delete it after verification — it contains plaintext secrets.

Step 5: Verify with preview, not up

Run pulumi preview. It should read remote state and report no unexpected diff (for a migrated stack, ideally zero changes). Then confirm state landed in S3:

aws s3 ls s3://my-org-pulumi-state/my-project/.pulumi/stacks/

Only after preview is clean, run pulumi up once. If you verified locking in step 3, you can test it by starting an update in one terminal and attempting a second — it should be rejected or serialized per the documented behavior. Do not run this concurrency test against production.

Recovery options

• Before any risky operation: pulumi stack export --file backup-$(date +%F).json. This is your fastest restore point.
• After drift: pulumi refresh reconciles state with actual cloud resources; follow with pulumi preview to confirm no unintended changes remain.
• Corrupted or lost state: restore from an export via pulumi stack import, or download a prior object version from S3 (versioning pays off here) and import that. Practice this in a disposable stack first.
• Stuck update: pulumi cancel to release the lock and mark the update complete, then refresh.

Never hand-edit state objects in S3 — the JSON contains dependency graphs and secret envelopes that are easy to break silently.

Limitations

You own durability and access control; enable bucket versioning and consider replication for critical state. Locking semantics depend on CLI version, so re-verify after upgrades. Secrets in state are encrypted with your stack's passphrase or KMS key — losing that key means losing the ability to decrypt secrets even if state survives.