Choosing Pulumi State Backend: Managed Service vs Self‑Hosted for Regulated Multi‑Team IaC

Decision and constraints

When multiple teams share Pulumi infrastructure code in a regulated setting, the state backend must satisfy:

• SOC 2 auditability and immutable audit logs
• Data‑residency or air‑gap requirements
• High availability for concurrent pulumi up operations
• Secure handling of secrets (state encryption)
• Predictable operational cost

Comparison of supported options

Feature	Pulumi Service (managed)	Self‑hosted (S3‑compatible + lock store)
Setup effort	Minimal – create organization, generate access token	Provision object storage, lock DynamoDB/SQLite, configure encryption passphrase
Audit & compliance	Built‑in audit logs, SOC 2 attestation, RBAC	Depends on your storage/logging solution; you must implement and attest
Data residency	Data resides in Pulumi‑managed regions (US/EU/APAC)	Full control – can be on‑prem or in a specific cloud region
Availability & locking	Managed locking service with automatic retries	You must ensure lock store HA; loss blocks updates
Cost	Subscription fee per user + optional add‑ons	Pay only for storage/lock resources; operational overhead adds cost
Secret handling	State encrypted at rest by Pulumi; optional secret providers	Client‑side encryption with a passphrase; you manage rotation and vaulting

Trade‑offs summary

• Managed backend reduces operational burden, provides ready‑made locking, backups, and UI collaboration, but introduces vendor dependency, required internet egress, and ongoing subscription cost.
• Self‑hosted backend satisfies strict data‑residency or air‑gap policies and removes vendor lock‑in, yet you must operate storage HA, ensure lock consistency, manage backups/restores, and safeguard the encryption passphrase.

Concrete validation steps

Use the following commands to test each backend in an isolated environment. Replace placeholders with your actual values.

1. Initialize a stack with the managed backend

# Run in a CI agent or developer workstation with Pulumi CLI installed
# Required permission: ability to create Pulumi access tokens in your organization
pulumi login https://app.pulumi.com   # ensures CLI talks to the service
pulumi stack init dev‑managed   # creates a new stack
pulumi config set pulumi:serviceToken  --secret
# Verify backend is being used
pulumi preview   # should show "Fetching state from Pulumi Service"

2. Initialize a stack with a self‑hosted backend

# Prerequisites: S3‑compatible bucket (e.g., MinIO) and DynamoDB lock table
# Required permission: read/write on the bucket and lock table
pulumi login s3://my‑pulumi‑state-bucket   # points CLI to the bucket
pulumi stack init dev‑selfhosted
pulumi config set aws:region us‑west-2
pulumi config set pulumi:stateBackend 
pulumi config set pulumi:lockBackend 
pulumi config set pulumi:passphrase  --secret
# Verify backend is being used
pulumi preview   # should indicate "Loading state from S3"

3. Test concurrent update locking

In two separate terminals, run:

pulumi up --yes   # first terminal
# In second terminal, run the same command while the first is still pending
pulumi up --yes   # should output "Attempting to acquire lock…" and wait or fail with a clear lock‑acquisition error

4. Backup and restore drill (self‑hosted only)
# Backup: copy state object and lock record
aws s3 cp s3://my-pulumi-state-bucket/dev-selfhosted/state.json ./state-backup.json
aws dynamodb get-item --table-name my-pulumi-lock-table --key '{"LockID":{"S":"dev-selfhosted"}}' > lock-backup.json

# Simulate failure: delete the original state
aws s3 rm s3://my-pulumi-state-bucket/dev-selfhosted/state.json

# Restore: put back the objects
aws s3 cp ./state-backup.json s3://my-pulumi-state-bucket/dev-selfhosted/state.json
aws dynamodb put-item --table-name my-pulumi-lock-table --item file://lock-backup.json

# Verify drift detection works
pulumi refresh   # should show no changes if restore was correct

Practical checks and risks

  
Permission check: Ensure the identity running Pulumi has s3:GetObject, s3:PutObject, and lock‑table GetItem/PutItem rights.
  
Risk of passphrase loss: If the passphrase for self‑hosted encrypted state is lost, recovery is impossible. Store it in a sealed vault with MFA‑protected access.
  
Risk of lock store downtime: An unavailable DynamoDB (or SQLite) lock table will cause pulumi up to hang or fail with lock‑acquisition errors. Monitor lock‑table health and configure alerts.
  
Verification of audit logs: For the managed backend, review pulumi audit log or the organization’s activity page to confirm that state reads/writes are logged per SOC 2 requirements.