Architecting Secure State and Secrets Management in Pulumi

The Problem: State Leakage and Secret Exposure

Infrastructure as Code (IaC) tools must track the current state of deployed resources to calculate diffs for updates. When using Pulumi, this state file often contains sensitive data—such as database passwords or API keys—that are returned by cloud providers during resource creation. If this state is stored in plaintext or managed with overly permissive access, a compromise of the storage backend results in a full compromise of the infrastructure credentials.

The goal is to ensure that no plaintext secrets exist in version control, state backends, or CI/CD logs, while maintaining the ability to reproduce environments reliably.

Smallest Suitable Design

For most teams, the most secure starting point is a design that separates environment configurations and leverages a dedicated secrets provider. This minimizes the blast radius of a single compromised stack.

• Program Structure: One Pulumi program per environment (e.g., staging, production) using TypeScript or Python.
• State Backend: Use a private object store (AWS S3, GCP GCS, or Azure Blob Storage) or Pulumi Cloud. If using a self-managed backend, enable versioning and object locking to prevent accidental deletion or corruption.
• Secrets Management: Use the built-in Pulumi secrets provider or a cloud KMS (Key Management Service) to encrypt configuration values before they are written to the state file.

Trust and Data Boundaries

Security in Pulumi depends on where the decryption key resides and who can access the state backend.

The Encryption Boundary

Config secrets are encrypted on the local machine or CI runner before being uploaded. Plaintext values exist only in the process memory of the Pulumi CLI during execution. The state backend stores only the ciphertext.

The Access Boundary

The state backend must be restricted via IAM policies. Only the specific deployment identity (e.g., a GitHub Actions OIDC role) and a limited number of "break-glass" administrators should have read/write access. The CI runner holds the decryption key—either via a passphrase or a KMS grant—only for the duration of the pulumi up or pulumi preview command.

Operational Implementation

To implement this design, use the --secret flag when setting configuration values. This ensures the value is encrypted using the stack's provider.

# Run this on a trusted terminal with appropriate cloud credentials
# This encrypts the value before it ever hits the state file
pulumi config set --secret dbPassword "super-secret-password"

Verification and Checks

To verify that secrets are not leaking into the state, perform a manual state export in a sandbox environment:

1. Run pulumi stack export > state.json.
2. Search the state.json file for the plaintext value of your secret.
3. Confirm the value appears as a ciphertext string (e.g., "urn:pulumi:{"secret":...}").

CI/CD Guardrails

Integrate the following checks into your pipeline:

• Read-Only Previews: Run pulumi preview on every Pull Request using a read-only IAM role. This prevents the CI from making unauthorized changes while validating the plan.
• Human Gating: Require a manual approval step before executing pulumi up in production.
• Backend Auditing: Enable access logging on the S3/GCS bucket to alert on any access requests originating from outside the CI/CD runner's IP range.

Failure Modes and Recovery

Failure Scenario	Impact	Mitigation/Recovery
Loss of KMS Key/Passphrase	Existing encrypted config is unrecoverable.	Rotate secrets in the cloud provider, update config with a new key, and re-run pulumi up.
Concurrent State Updates	State corruption or "lost updates."	Use a backend that supports locking (e.g., Pulumi Cloud or S3 with DynamoDB locking).
Verbose Logging	Plaintext secrets leaked to CI logs.	Avoid -v or --debug flags in production pipelines; use Pulumi's built-in secret masking.

When to Evolve the Design

The "smallest suitable design" should be revisited when the following conditions are met:

• Compliance Requirements: If your industry requires Customer Managed Keys (CMK) for all data at rest, migrate from the default Pulumi provider to a cloud-native KMS provider.
• Organizational Scale: When multiple teams share a single environment, move toward Pulumi ESC (Environments, Secrets, and Configuration) to manage access tokens and secrets centrally rather than per-stack.
• Policy Enforcement: If you need to prevent specific resource types (e.g., preventing public S3 buckets), implement CrossGuard policy packs to enforce guardrails during the preview phase.