Configure Terraform Remote State on Amazon S3 with DynamoDB Locking

The Problem with Local Terraform State

When you run Terraform with a local backend, the state file sits on your workstation. If two engineers run terraform apply against the same infrastructure from different machines, Terraform has no shared view of reality. The second apply can overwrite the first, causing resource drift, orphaned resources, or corrupted state. A remote backend solves this by storing state in a shared location and serializing operations with a lock.

Desired Outcome

By the end of this guide, your Terraform state will live in an Amazon S3 bucket with versioning enabled, and every operation will acquire a lock from a DynamoDB table before proceeding. This prevents concurrent writes and gives you a recovery path through S3 object versioning.

Prerequisites

• An AWS account with permissions to create and manage S3 buckets and DynamoDB tables.
• Terraform CLI version 1.0 or later installed. You can verify with terraform version.
• AWS credentials configured for the Terraform CLI. This is typically done through the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, a shared credentials file at ~/.aws/credentials, or an IAM role attached to your compute environment.
• An existing Terraform configuration with a local state file that you want to migrate, or a new project starting from scratch.

Step 1: Create the S3 Bucket and Enable Versioning

Run the following commands in a terminal where your AWS credentials are available. Replace my-terraform-state-bucket with a globally unique bucket name.

aws s3api create-bucket --bucket my-terraform-state-bucket --region us-east-1

aws s3api put-bucket-versioning --bucket my-terraform-state-bucket --versioning-configuration Status=Enabled

Versioning keeps every version of the state object. If a bad apply overwrites the state, you can retrieve a prior version instead of rebuilding from scratch.

Step 2: Create the DynamoDB Lock Table

Terraform uses a DynamoDB table to store lock metadata. The table must have a primary key named LockID with a string type. Create it with:

aws dynamodb create-table \
  --table-name my-terraform-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

PAY_PER_REQUEST billing avoids the need to provision read/write capacity units, which reduces the risk of throttling during bursts of lock acquisitions. If you expect very high concurrency, you can switch to provisioned capacity and monitor throttling with CloudWatch.

Step 3: Configure the Backend Block

Add a backend block to your Terraform configuration. If you are migrating an existing project, place this in the same directory as your .tf files. The key value acts as the file path inside the bucket.

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "prod/network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "my-terraform-locks"
    encrypt        = true
  }
}

Setting encrypt = true enables server-side encryption with Amazon S3 managed keys (SSE-S3). If your organization requires customer-managed keys, you can add kms_key_id instead, but that requires additional KMS permissions.

Step 4: Initialize and Migrate State

Run initialization from the directory containing your backend configuration:

terraform init

If Terraform detects an existing local state file, it prompts you to copy it to the new backend. Confirm the migration. If you are starting a new project, initialization simply configures the backend.

Step 5: Apply Least-Privilege IAM Permissions

The credentials Terraform uses need specific permissions. A minimal policy for the state bucket and lock table looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::my-terraform-state-bucket/prod/network/terraform.tfstate"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/my-terraform-locks"
    }
  ]
}

Adjust the account ID and resource ARNs to match your environment. Note that s3:DeleteObject is required because Terraform deletes and rewrites the state object during apply.

Expected Checks

After terraform init, the output should include a message confirming the S3 backend was configured successfully. To verify locking, open two terminals in the same Terraform directory. Run terraform apply in the first terminal. While it is running or immediately after, run terraform apply in the second terminal. The second command should fail with an error indicating the state lock could not be acquired.

You can inspect the state object and lock table directly:

aws s3api head-object --bucket my-terraform-state-bucket --key prod/network/terraform.tfstate

aws dynamodb get-item --table-name my-terraform-locks --key '{"LockID":{"S":"my-terraform-state-bucket/prod/network/terraform.tfstate-md5"}}'

The head-object response should show "VersionId" and encryption metadata. The DynamoDB item contains an Info attribute with operation details and may include a TTL attribute for automatic cleanup after a stale period.

Recovery Options

Stale Locks

If a Terraform process is interrupted, the lock item may remain in DynamoDB. Before removing it, confirm no other Terraform process is running against that state. Then delete the item:

aws dynamodb delete-item --table-name my-terraform-locks --key '{"LockID":{"S":"my-terraform-state-bucket/prod/network/terraform.tfstate-md5"}}'

Removing a lock while another operation is active can cause state corruption. Use this only when you are certain the previous run has terminated.

Restoring a Previous State Version

Because S3 versioning is enabled, you can list object versions and download an earlier state file:

aws s3api list-object-versions --bucket my-terraform-state-bucket --prefix prod/network/terraform.tfstate

Download the desired version, inspect it locally, and then upload it back as the current version if recovery is needed. Always validate the state with terraform plan before running another apply.

Limitations and Considerations

• DynamoDB throttling: Extremely high-frequency lock operations can exhaust provisioned capacity. On-demand billing mitigates this, but very large teams should still monitor CloudWatch metrics.
• Bucket deletion protection: S3 versioning alone does not prevent bucket deletion. Consider adding an S3 bucket policy that denies s3:DeleteBucket and enabling MFA delete if your compliance requirements demand it.
• Backend initialization is directory-scoped: Running terraform init from a different working directory with the same backend key will attach to the same state file. Use distinct key values for separate environments or workspaces.
• Encryption scope: The encrypt setting protects the state object at rest. It does not encrypt the DynamoDB lock table. If required, enable encryption at rest for DynamoDB separately using an AWS owned key or customer managed key.

Practical Verification

Run terraform plan after initialization. If the backend is configured correctly, Terraform reads state from S3 without prompting for backend configuration. The plan output should reflect the resources tracked in the remote state. If you see errors about backend configuration or state locks, revisit the IAM policy and confirm the DynamoDB table name and bucket region match your backend block exactly.