DEV Community

loading...

Rolling back your Terraform Remote State in S3

Shannon
SRE in Portland. Mostly writing on topics that pique my interest and can be referenced later on by myself. Currently on a post-pandemic work break :)
・2 min read

While I hope this is a rare occurrence, I've run into this a few times where I had to rollback Terraform state to a previous version in AWS S3. Why not just keep it in the newer version? Well, some of our CI/CD pipelines used an older version of Terraform (say 0.12), and while none of the resource states actually changed, our code would no longer run after a developer ran something locally on their computer with a newer version of Terraform (say 0.14).

So! Here's a simple example on how to rollback state.

Note: you must have versioning enabled on the S3 bucket. If it isn't, PLEASE do that for all places that Terraform State is stored :p

Setup

I'm going to set up this up completely fresh with a new remote state object in S3 with the following Terraform code:

terraform {
  backend "s3" {
    bucket  = "shannon-terraform"
    key     = "dev.to-example/terraform.tfstate"
    region  = "us-west-2"
    profile = "shannon"
  }

  required_providers {
    null = {
      source  = "hashicorp/null"
      version = "~> 3.1"
    }
  }
}

resource "null_resource" "fake1" {
  count = 1
}
Enter fullscreen mode Exit fullscreen mode

So, I terraform apply -auto-approve'd this. Next, I'm ticking up the count to 2 and applying that as well. Thus, I have two different versions of Terraform State saved in that bucket path.

Now, we should be able to see these with the S3 api: aws s3api list-object-versions --bucket shannon-terraform --prefix dev.to-example/terraform.tfstate --output json --profile shannon | jq -r ".Versions[].LastModified"

In my case, I see two modifications:

2021-06-23T22:17:02+00:00
2021-06-23T22:16:28+00:00
Enter fullscreen mode Exit fullscreen mode

Next up, I'm gonna grab a copy of my previous Terraform State locally in my current working directory: aws s3api get-object --bucket shannon-terraform --key dev.to-example/terraform.tfstate --version-id [VersionID] [local_copy_name] --profile shannon If you're wondering where the VersionID comes from, it should be in the output of the above list-object-versions command, but it won't show up without removing the jq parsing.

Finally, let's push up that local backup into the S3 path: aws s3 cp ./[local_copy_name] s3://shannon-terraform/dev.to-example/terraform.tfstate --profile shannon. Re-run the above command for listing objects, and you should now see another version!

Finally, you can confirm the update with terraform state list. In my case, I now only see one null_resource again after rolling the state back to before creating a second one.

Please be careful in the use cases here! Getting Terraform State out of sync with actual infrastructure can be disastrous. This should be a niche case for ever using this.

Discussion (0)