Migrating from Terraform Cloud to Amazon S3 and DynamoDB: A Guide

Terraform’s recent move to the resources under management pricing model has frustrated a lot of users. As one user on this reddit thread remarked:

The pricing on Terraform literally just jumped to almost same as we are paying for the AWS resources it manages.We’re on the free plan, with only 2 devs — were looking at moving to the paid plan this month which was going to be $20/user/month — but now we’re looking at a ~ $250 monthly bill for our dev, test and prod environments. We’re only paying $300 for the AWS resources.

Migrating your backend from Terraform Cloud to an infrastructure based on Amazon S3 and DynamoDB may help save you some $$ and help extend that end of runway.

It involves configuring Terraform to use these services for remote state storage. Here’s a guide on how to perform this migration:

• Prepare the infrastructure:

Create an S3 bucket: Log in to the AWS Management Console and create a new S3 bucket to store your Terraform state files. Choose a unique name for the bucket.

Create a DynamoDB table: In the AWS Management Console, create a new DynamoDB table to manage Terraform state locking. Provide a table name and a primary key (e.g., “LockID” of type String).

• Update Terraform configuration:

Open your Terraform configuration files (e.g., main.tf, backend.tf).

Remove the existing Terraform Cloud backend configuration.
Add a new backend configuration to use S3 and DynamoDB. Here’s an example:

terraform {
  backend "s3" {
    bucket         = "<your-s3-bucket-name>"
    key            = "terraform.tfstate"
    region         = "<aws-region>"
    dynamodb_table = "<your-dynamodb-table-name>"
  }
}

NOTE: Replace <your-s3-bucket-name> with the name of the S3 bucket you created in step 1, <aws-region> with the appropriate AWS region, and <your-dynamodb-table-name> with the name of the DynamoDB table you created.

• Initialize the backend:

Open a terminal or command prompt in the directory containing your Terraform configuration files.
Run terraform init to initialize the new backend configuration and download any necessary provider plugins.

• Migrate the existing state:

(Hashicorp Help Center Docs)

If you have an existing Terraform state in Terraform Cloud, you’ll need to download it and migrate it to the new backend.
Use the terraform state pull command to download the current state from Terraform Cloud.

Run terraform init -reconfigure to reconfigure Terraform with the new backend and upload the state to S3 and DynamoDB.

• Test the new backend

Run any Terraform commands (e.g., terraform plan, terraform apply) to ensure the new backend is working correctly.
Verify that the state files are stored in the S3 bucket and that the DynamoDB table is being used for state locking.

• Clean up:

If you’re confident that the migration was successful, you can remove the old Terraform Cloud backend configuration and associated state files.

• Migrate your Terraform CI/CD:

There are plenty of options for cheaper alternatives to Terraform Cloud — This article on Terraform automation and collaboration software speaks about it in detail.

There are some DIY Github Actions that can be added as steps to your own workflow files. Here is an example of one such tool.

Disclosure: We’re building Digger — an open source GitOps tool for Terraform that reuses the async jobs infrastructure with compute, orchestration, logs, etc of your existing CI/CD. Do feel free to try it out!