Managing S3 bucket for Terraform backend in the same configuration

What is Terraform backend?

A Terraform backend is a place where Terraform uses to store its state. Terraform supports various types of backend. The default local backend stores the state files in the local filesystem. Local backend is useful for creating resources for testing purposes in a "quick and dirty" manner. However, it is common practice to use backends that store states in remote services such as Amazon S3 (S3) when working on production resources as it supports state locking and versioning. Managing S3 bucket for Terraform backend in the same configuration

The chicken-and-egg problem

The following is an example of using an S3 bucket as Terraform backend. Notice that we need to specify the name of the S3 bucket.

terraform {
  required_version = "~> 1.1.4"
  backend "s3" {
    bucket = "terraform-s3-backend-pmh86b2v"
    region = "ap-northeast-1"
    key    = "terraform.tfstate"
  }
}

The above means that we need to have the S3 bucket for the backend before writing Terraform configurations to manage other resources. What if we also want to create the S3 bucket for the backend using Terraform? A straightforward approach is to manage the S3 bucket for the backend in a separate Terraform configuration that uses a local backend. We want to show how to manage that in the same configuration as other resources in this article.

The solution

Let’s take a look at the following Terraform configuration. We are going to create two S3 buckets: one for the backend (terraform-s3-backend-pmh86b2v) and another one is the actual bucket that we need for our project (my-project-...) (If you plan to follow this “tutorial,” please change the bucket name as they need to be globally unique). Notice that at this point, we are using a local backend.

terraform {
  required_version = "~> 1.1.4"
}

provider "aws" {
  region = "ap-northeast-1"
}

resource "aws_s3_bucket" "backend" {
  bucket = "terraform-s3-backend-pmh86b2v"
  acl    = "private"
}

resource "aws_s3_bucket" "main" {
  bucket_prefix = "my-project-"
  acl           = "private"
}

We start by only creating the S3 bucket (terraform-s3-backend-pmh86b2v) for the backend using the target flag -target. We can see that the command above also creates a state file (terraform.tfstate) in our local directory.

$ terraform plan -target=aws_s3_bucket.backend -out=/tmp/tfplan
$ terraform apply /tmp/tfplan

Then, we configure Terraform to use our new S3 bucket as its backend by changing the backend block.

diff --git a/main.tf b/main.tf
index c6d846b..19547ae 100644
--- a/main.tf
+++ b/main.tf
@@ -1,5 +1,10 @@
 terraform {
   required_version = "~> 1.1.4"
+  backend "s3" {
+    bucket = "terraform-s3-backend-pmh86b2v"
+    region = "ap-northeast-1"
+    key    = "remote_terraform.tfstate"
+  }
 }

 provider "aws" {

When we run terraform plan, Terraform will detect the changes in the backend and prompt us to reinitialize. Run recommended the command terraform init to make the switch.

$ terraform plan -out=/tmp/tfplan
╷
│ Error: Backend initialization required, please run "terraform init"
│
│ Reason: Initial configuration of the requested backend "s3"
...
$ terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes
...

The migration copies the state file to S3. We can confirm this via the AWS CLI or from AWS Management Console.

$ aws s3 ls terraform-s3-backend-pmh86b2v
2022-01-31 19:46:06       1734 remote_terraform.tfstate

After the migration is complete, we can use Terraform (plan, apply, etc.) as usual.

How to migrate from the S3 backend to the local backend?

As Terraform will detect the changes in the backend, change the backend block back to local to migrate the state file to a local directory.

diff --git a/main.tf b/main.tf
index 19547ae..e5cac0b 100644
--- a/main.tf
+++ b/main.tf
@@ -1,10 +1,6 @@
 terraform {
   required_version = "~> 1.1.4"
-  backend "s3" {
-    bucket = "terraform-s3-backend-pmh86b2v"
-    region = "ap-northeast-1"
-    key    = "remote_terraform.tfstate"
-  }
+  backend "local" {}
 }

 provider "aws" {

When running terraform plan, Terraform will notice the change in backend and prompt us to run

$ terraform init -migrate-state

Since Terraform will not remove the state files from the previous backend, we have to remove the contents in the S3 bucket before removing the bucket from the Terraform configuration. Or, we can configure the backend bucket with force_destroy = true before removing it from the Terraform.

Final remarks

As shown above, Terraform can automatically detect changes in the backend configuration. We can use this feature to help solve the chicken-and-egg problem when using the S3 bucket as Terraform backend. Please find the sample code in this article at https://github.com/shihanng/terraform-s3-backend.

• Cover image by Taylor Vick, licensed under Unsplash License