Preventing Terraform State Corruption with Remote Locking

The concurrency problem

Terraform records the identity and attributes of every managed resource in a state file. When two engineers—or two CI jobs—run terraform apply against the same configuration at the same time, each process reads the current state, calculates a plan, and writes back an updated file. The second write overwrites the first, so the state no longer matches the real infrastructure. The next run may try to recreate resources that already exist, or worse, destroy them.

A local file backend offers no coordination mechanism, so it cannot prevent this race. A remote backend that implements locking solves the problem by serialising write access: only one run may hold the lock at a time.

How remote state locking works

Backends that support locking—such as Terraform Cloud, Terraform Enterprise, Azure Blob Storage with a lease, and AWS S3 paired with DynamoDB—create a small lock object alongside the state file. When a run begins, Terraform attempts to acquire that lock:

1. The CLI requests the lock from the backend.
2. If the lock is free, the backend records the lock metadata and returns success.
3. The run proceeds through plan and apply.
4. On successful completion, Terraform releases the lock automatically.

If another run arrives while the lock is held, it blocks until the lock disappears or fails after a configurable timeout. A crashed run can leave a stale lock behind. Some backends implement a time-to-live (TTL) so the lock expires on its own; otherwise an operator must clear it manually with terraform force-unlock.

Worked example: Terraform Cloud workspace

The following steps show a minimal configuration that enables locking for a team sharing a workspace. Run the commands from a workstation that has the Terraform CLI (version 1.5 or newer) and network access to app.terraform.io. You need write permission on the target workspace.

# 1. Add a cloud backend block to your root module (main.tf)
terraform {
  cloud {
    organization = "my-org"
    workspaces {
      name = "shared-infra"
    }
  }
  required_version = ">= 1.5"
}

# 2. Initialise the backend
terraform init

After init, the remote state and lock objects are created. Verify that the workspace appears in the Terraform Cloud UI before proceeding.

# 3. Start two applies in separate terminals
# Terminal A
terraform apply -auto-approve

# Terminal B (started a few seconds later)
terraform apply -auto-approve

Terminal B will not proceed while Terminal A holds the lock. Depending on the backend and CLI version, it will either wait for the lock to be released or eventually exit with a lock timeout error. Once Terminal A finishes successfully, the lock is released and Terminal B can continue. If Terminal A crashes, the lock may remain; you can clear it with:

# Requires appropriate permissions on the workspace
terraform force-unlock -force LOCK_ID

Replace LOCK_ID with the identifier shown in the workspace UI. Use -force only after confirming that no other run is active; otherwise you risk corrupting state.

Trade-offs and limitations

• Cost and latency: Managed backends add a subscription fee and a network round-trip for every lock acquisition. For small teams, a self-hosted S3 + DynamoDB backend may be cheaper but requires operational overhead.
• Stale locks: A crash or network partition can leave a lock that never expires unless the backend implements a TTL. Teams must monitor the workspace or script periodic cleanup.
• Backend support: Not every backend implements locking. Local files, plain S3 without DynamoDB, and some third-party stores provide no safety—concurrent runs will still race.
• Throttling: High request rates from many parallel CI pipelines can hit API limits on the backend, causing plan/apply delays. Consider a dedicated workspace per environment or a queueing layer.

Actionable checklist

1. Choose a backend that documents lock support (Terraform Cloud, Azure Blob with lease, S3 + DynamoDB).
2. Add the backend block to your configuration and run terraform init.
3. Verify locking works: start two applies in separate shells and confirm the second waits or fails with a timeout.
4. Enable a lock TTL where available, or configure a monitoring alert for locks older than a threshold such as 30 minutes.
5. Document the force-unlock procedure and restrict it to privileged operators.
6. Review backend pricing and latency; if costs are prohibitive, evaluate a self-managed S3 + DynamoDB setup with the same locking guarantees.

Following these steps gives you a reproducible safety net: every state change is serialised, stale locks are visible, and the team can run Terraform confidently in parallel.