Streamline Your Terraform Backend: Ditching DynamoDB with S3 Native State Locking

For years, Terraform users on AWS managed state files in an S3 bucket and relied on a DynamoDB table for locking. This two-service approach added complexity, cost, and operational overhead. With Terraform 1.10 and AWS's native S3 object lock feature, you can now simplify your backend—eliminating DynamoDB entirely. Below, we answer common questions about this new capability, from setup to migration and troubleshooting.

Why was DynamoDB originally needed for Terraform state locking?

Terraform state files track your infrastructure's current state. When multiple users or CI/CD pipelines run terraform apply simultaneously, the state file can become corrupted if both processes modify it. DynamoDB provided a distributed locking mechanism: before applying, Terraform would write a lock record to a DynamoDB table. If another process tried to apply, it would see the lock and wait. This prevented concurrent modifications but introduced extra complexity—teams needed to provision and manage a separate DynamoDB table for every environment, and stuck locks required manual deletion of records.

Streamline Your Terraform Backend: Ditching DynamoDB with S3 Native State Locking — Source: www.freecodecamp.org

What exactly is S3 native state locking and how does it work?

Announced in November 2024 and supported in Terraform 1.10, S3 native locking uses AWS S3's object lock feature to provide lease-based locking directly on the state file. When Terraform wants to apply changes, it requests a write lock on the state object in S3. S3 grants a temporary lease, and Terraform updates the file. If another process tries to lock the same object, S3 denies the request until the lease expires or is released. This eliminates the need for a separate DynamoDB table—the S3 bucket itself handles both storage and locking, simplifying the backend to just one resource.

How does S3 native locking compare to the old S3 + DynamoDB approach?

The key differences are:

Complexity: Old setup required two services (S3 + DynamoDB); new setup uses only S3.
Cost: DynamoDB on-demand billing can add up across multiple teams/environments; S3 native locking has no additional charges beyond standard S3 costs.
Operations: Stuck locks no longer require manual DynamoDB deletions—S3 leases expire automatically or can be released via S3 APIs.
Setup: New projects need only one bucket with object lock enabled; migrations require enabling object lock on an existing bucket and updating backend config.

How do I set up S3 native locking from scratch for a new project?

Start by creating an S3 bucket with versioning and default encryption enabled. Crucially, you must also enable object lock on the bucket—this can only be done at bucket creation time. Then, configure your Terraform backend block to use the new use_lockfile option:

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    use_lockfile   = true
  }
}

Run terraform init to set up the backend. Terraform will now use S3's native locking automatically. Verify by checking that the state file has an active object lock lease when an apply is running.

How do I migrate my existing Terraform backend from DynamoDB to S3 native locking?

First, verify your current setup: your S3 bucket must have versioning enabled. Next, enable object lock on the bucket—note that this can only be done on existing buckets if they were created with the ObjectLockEnabledForBucket parameter, or you may need to create a new bucket and migrate the state file. If allowed, you can use the AWS CLI: aws s3api put-object-lock-configuration --bucket your-bucket --object-lock-configuration DefaultRetention.... Then update your Terraform backend configuration to remove the DynamoDB table reference and add use_lockfile = true. Run terraform init -reconfigure to apply the changes. Finally, clean up the DynamoDB table once you've confirmed locking works.

How can I verify that locking is working correctly?

To test, open two terminals. In the first, run terraform apply and let it pause (e.g., by adding a local-exec with a sleep). In the second terminal, try to run another terraform apply on the same state. You should see an error message similar to: Error: acquiring state lock: resource temporarily unavailable. This confirms S3 native locking is engaged. After the first apply completes, the lock releases and the second apply should succeed. Additionally, you can inspect the state file in the S3 console—during a lock, it will show a temporary legal hold or retention period.

What should I do if a lock gets stuck?

With DynamoDB, stuck locks required manual deletion of records. With S3 native locking, leases have a default timeout (typically a few minutes). If a Terraform process crashes or is killed, the lease will expire automatically. If you need to force release a lock, you can use the AWS CLI: aws s3api delete-object-legal-hold --bucket your-bucket --key your-state-file --version-id version-id. However, it's safer to wait or investigate the stuck process first. As a best practice, always ensure your Terraform pipelines have proper timeout handling to avoid orphaned locks.

What are the security best practices for my state bucket after migration?

Even though DynamoDB is gone, securing your S3 bucket remains critical. Apply the following measures:

Bucket policies: Restrict access to the bucket using IAM roles and conditions (e.g., only allow access from approved VPC endpoints).
Encryption: Enable default SSE-S3 or SSE-KMS encryption.
Versioning: Keep versioning enabled to recover from accidental deletion or corruption.
Object lock: Use default retention settings to prevent state file overwrites without proper authorization.
Monitoring: Enable S3 server access logs or AWS CloudTrail to audit all state file operations.

By following these practices, your Terraform backend remains secure and simple.

Tags: