Per-Stack Update Locking: Does Pulumi Queue Concurrent Runs or Fail Immediately?

Question

When a stack is being updated, Pulumi’s Cloud backend automatically acquires a per‑stack lock. On self‑managed backends (file, S3, GCS, Azure Blob) this lock is not enforced, so two pulumi up processes can race on the state file. The CLI offers a --parallel flag to control how many resource operations run concurrently, but the interaction between this flag, the lock mechanism, and the backend type is not fully documented.
In practice, an in‑progress run may either return a busy/lock error immediately or queue the second run until the first completes, depending on version and backend. This uncertainty complicates CI pipelines that trigger parallel deployments and can lead to state corruption or wasted retries.
Key constraints include:

Self‑managed backends lack automatic per‑stack locking.
The default --parallel value is derived from the machine’s CPU count and varies by CLI version.
Providers run in separate plugin processes; higher parallelism increases concurrent gRPC calls.

Unresolved decisions:

Should a concurrent update be queued or fail immediately when a lock is held?
Which mutual‑exclusion strategy (CI‑level lock, migration to Pulumi Cloud, or custom lock file) is most reliable for self‑managed backends?

How does the current default --parallel setting affect the likelihood of encountering lock errors during simultaneous updates?
What is the exact sequence of events when a second pulumi up is launched while the first is still running on a self‑managed backend?

Tasadduq Burney · Answer

Short answer
On a self‑managed backend (file, S3, GCS, Azure Blob) Pulumi does not queue concurrent runs. The second pulumi up will either:

return a busy/lock error immediately (newer CLI releases), or
  proceed without error and potentially overwrite the state file (older releases or backends with eventual consistency).

In either case the --parallel flag does not coordinate across separate CLI processes; it only throttles resource operations inside a single run.
Why this matters for CI
CI pipelines that launch parallel deployments can suffer from:

immediate failures that need retry logic, or
  state corruption if the second run writes over the first’s changes.

Both scenarios are prevented if you either migrate to Pulumi Cloud (which enforces per‑stack locks) or add a CI‑level lock (e.g., a DynamoDB entry, Redis key, or a simple file lock on a shared network filesystem).
Verifying the behavior in your environment

Check the CLI version:pulumi version
  Run two pulumi up commands in parallel against the same stack on a self‑managed backend (e.g., a local file store). Use --parallel 1 to isolate resource ops if you want deterministic output.
  Observe the exit status of the second run:
    
      0 – it succeeded; check the state file timestamp to see if it overwrote the first run.
      Non‑zero with a message containing busy or lock – it failed immediately.

If the second run succeeded, run pulumi state list and compare the resource counts to the first run to confirm whether changes were lost.
  Repeat the test on Pulumi Cloud to confirm that the second run is queued and the first completes first.

What to do next
For reliable CI, consider one of the following:

Switch the stack to Pulumi Cloud to get built‑in locking.
  Add a lightweight CI‑level lock (e.g., a DynamoDB entry keyed by stack name) that the pipeline checks before invoking pulumi up.
  If you must stay on a self‑managed backend, enforce single‑process updates by serializing pipeline jobs.

Missing diagnostic detail
To tailor the recommendation precisely, could you share the exact Pulumi CLI version you are using? The lock‑error semantics changed in the 2024.3 release, so the outcome may differ between 3.45 and 3.48.

Per-Stack Update Locking: Does Pulumi Queue Concurrent Runs or Fail Immediately?

1 answer

Short answer

Why this matters for CI

Verifying the behavior in your environment

What to do next

Missing diagnostic detail

Your perspective belongs here.

0 question comments