DEV Community

globart
globart

Posted on • Edited on

S3 native state locking in Terraform

Since the Terraform 0.5.0 release from May 2015th we've been able to store our state on S3 buckets.

But in order to ensure it's consistency, we've had to use state locking using DynamoDB table. Although the DynamoDB-associated costs are negligable, it is still nonetheless another resource, that you have create manually, which could prove to be pretty annoying if you are managing large infrastructure and/or many projects at once.

The DynamoDB was needed due to the fact that S3 only had eventual consistency, meaning that if you read a file some time after it was written, you would get its latest version. But if you attempted to read a file immediately after it was written, it wasn't guaranteed.
This was addressed when AWS introduced strong read-after-write consistency for S3 in December 2020

However, this didn't solve the issue entirely, because there was no build-in mechanism to check if an object exists, before creating it.
Fortunately, after another 4 years, Amazon introduced support for conditional writes in S3 in August 2024

These changes made it possible to start work on state locking without DynamoDB, which doesn't require any additional resources, apart from the bucket itself.
After a couple of months, S3 native state locking was introduced in Terraform 1.10.0 in November 2024
While similar discussion exists in OpenTofu repo since September 2023, at the time of writing this article, there was no equivalent solution developed. But I hope it will be created in the coming months.

So, while previously your S3 backend configuration looked like this:

terraform {
  backend "s3" {
    bucket         = "example-bucket"
    key            = "path/to/state"
    region         = "us-east-1"
    dynamodb_table = "example-table"
  }
}
Enter fullscreen mode Exit fullscreen mode

Now it can look like this:

terraform {
  backend "s3" {
    bucket       = "example-bucket"
    key          = "path/to/state"
    region       = "us-east-1"
    use_lockfile = true
  }
}
Enter fullscreen mode Exit fullscreen mode

While terraform apply is going on, key.tflock file will be created in S3 bucket, which contains lock information, including a unique lock ID and other metadata.
key.tflock file
If another user tries to do terraform apply at the same time, Terraform will see that the key.tflock file already exists, so apply will fail.
After apply is completed, key.tflock file will be deleted.

Currently, while this feature is still experimental, use_lockfile argument is optional and defaults to false.
To support migration from older versions of Terraform which only support DynamoDB-based locking, it can be configured simultaneously with dynamodb_table argument. In a future minor version the DynamoDB locking mechanism will be removed.

Top comments (0)