A Comprehensive Guide on Migrating OCI Terraform State from Local to Remote Storage

Introduction

Infrastructure as Code (IaC) has revolutionized how organizations manage their cloud resources. Terraform, a popular IaC tool, enables developers and operations teams to define and provision infrastructure using declarative configuration files. When working with Oracle Cloud Infrastructure (OCI) and Terraform, managing the state file is a crucial aspect of your infrastructure workflow.

In this guide, we will explore the process of migrating your Terraform state from a local file system to an OCI Object Storage bucket. This migration enhances the security and scalability of your state management and allows for better collaboration among team members.

Why Migrate Terraform State to an Object Storage?

To begin with, it is important to thoroughly understand the differences between storing your Terraform state on a local file system versus on OCI Object Storage. While a local file system may seem convenient at first, it can lead to potential data loss, security breaches, and difficulty in scaling your infrastructure. Object Storage, on the other hand, offers a highly secure and scalable solution to store your Terraform state, ensuring that your data is always accessible and protected.

Migrating Terraform state from your local file system to an Object Storage generally offers several compelling benefits:

1. Enhanced Security: Storing your state in OCI Object Storage ensures that it's stored in a secure and redundant manner. You can take advantage of OCI's robust security features, including encryption and IAM (Identity and Access Management) policies, to control access.
2. Improved Collaboration: Object Storage enables seamless collaboration among team members by providing a centralized location for storing and sharing Terraform state. Multiple team members can access and modify the state file without the risk of conflicts.
3. Scalability and Reliability: OCI Object Storage is designed to scale automatically to accommodate your growing infrastructure. It offers high availability and durability, ensuring your Terraform state is always accessible.
4. Versioning and Auditing: Object Storage can be configured to enable versioning, allowing you to track changes to your Terraform state over time. This feature is invaluable for auditing and troubleshooting.

Prerequisites

Before you begin the migration process, ensure you have the following prerequisites in place:

1. OCI Account: You should have an active OCI account with the necessary permissions to create and manage Object Storage buckets.
2. Terraform Installed: Ensure that Terraform is installed on your local machine. You can download it from the official Terraform website (https://www.terraform.io/downloads.html).
3. OCI CLI: Install the Oracle Cloud Infrastructure Command Line Interface (OCI CLI) and configure it with the necessary credentials. You can follow the official documentation for installation instructions (https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm).

Prepare for the Migration Process

The migration process involves creating an Object Storage bucket, updating the Terraform configuration to specify the new backend configuration for storing the state in OCI Object Storage, initializing Terraform with the new backend configuration and migrating the existing local Terraform state to OCI Object Storage.

Create an Object Storage Bucket

The first step is to identify your namespace. In OCI, a namespace is a unique identifier that is used to create a unique name for each object in Object Storage. The namespace is automatically generated when you sign up for an OCI account and cannot be changed.

Retrieve your OCI Object Storage namespace using the following command:

namespace = oci os ns get

This will return a string that represents your namespace and set it as an environment variable in your CLI for easy reference in subsequent commands.

Next, create an Object Storage bucket in your OCI console, ensuring you choose a unique and meaningful name for your bucket. You can also create the bucket using the OCI CLI with the following command:

oci os bucket create --name <bucket-name> --namespace $namespace

Replace "bucket-name" with the actual bucket name and the namespace would be the one set from the previous command.

Use the following commands to migrate your existing local Terraform state to OCI Object Storage, including the additional commands for setting up authentication:

Create a Customer Secret Key

The customer secret key is a key generated for your user in OCI that you can use for authentication when interacting with OCI Object Storage. It consists of an access key ID and secret access key that you can use to authenticate and connect with your Object Storage bucket.

First export your email address as a variable:

export email=<your_email>

This command exports your console email address as a variable in the Command Line Interface (CLI) environment. It sets the variable email to your email address.

Verify the Exported Email:

echo $email

Then run the OCI CLI command to Query Your User Data from OCI IAM service to retrieve user data:

oci iam user list --query "data[?\"name\"=='$USERNAME']".{"name:\"name\",id:\"id\""} --output table

The -query parameter filters the data to find a user with a name matching the value in the $USERNAME variable and the resulting table includes the user's name and OCI User ID (OCIUID).

Export the Returned OCIUID as a Variable:

export OCIUID=<YOUR_OCIUID>

This command exports your OCI User ID (OCIUID) as a variable in the CLI environment. Replace the provided value with the OCIUID obtained from the previous command.

Finally, we can now create a customer secret key for your user using the OCI CLI. Create a customer secret key using the command below

oci iam customer-secret-key create --user-id $OCIUID --display-name 'key-tf-test' --query "data".{"AWS_ACCESS_KEY_ID:\"id\",AWS_SECRET_ACCESS_KEY:\"key\""} --output=table

Create a customer secret key for your user and display it in a tabular format. The -user-id parameter specifies your OCI User ID, and the -display-name parameter sets a name for the secret key. The -query parameter extracts the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY fields from the command's output and formats the output as a table.

Note down the result, we are going to use this for authentication when interacting with OCI blob storage.

Now, export AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as variables:

export AWS_ACCESS_KEY_ID=<YOUR_AWS_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_AWS_SECRET_ACCESS_KEY>

These commands export the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY obtained from the previous OCI CLI command as environment variables.

Update Terraform Configuration

In order to establish the connection between your Terraform workspace and the remote state stored in Object Storage, Terraform needs to be aware of the new backend configuration, which directs it to use OCI Object Storage for storing and managing the state of your infrastructure.

Inside your Terraform configuration file that defines the provider, typically a provider.tf file, you would locate or add a terraform block. This block is used to configure various aspects of Terraform's behaviour, including the backend configuration for storing the state. Here's a basic example of how you can configure the block:

terraform {
  backend "s3" {
    bucket         = "bucket-name"
    key            = "terraform.tfstate"
    region         = "your-region"
        endpoint       = "https://<bucket-name>.compat.objectstorage.<region>.oraclecloud.com"

    skip_region_validation      = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    force_path_style            = true
  }
}

Let's break down what each of these settings means:

• backend: Specifies the backend type. In this case, we are using "s3" because OCI Object Storage is S3-compatible.
• bucket: The name of the OCI Object Storage bucket where your Terraform state will be stored. Replace "-bucket-name" with the actual name of the bucket you created earlier.
• key: The name of the state file within the bucket. It's typically named "terraform.tfstate" but can be customized if needed.
• region: The OCI region where your Object Storage bucket is located. Replace "your-region" with the appropriate region identifier.

skip_region_validation, skip_credentials_validation, skip_metadata_api_check, and force_path_style: These settings control various validation and behaviour options for the backend configuration. Here's what each setting does:

• skip_region_validation: Skips validation of the region name, which is useful when using custom endpoints.
• skip_credentials_validation: Skips validation of credentials. Be cautious when using this option, as it can potentially lead to authentication issues.
• skip_metadata_api_check: Skips checking the metadata API for credentials. This can be useful when running Terraform in environments where the metadata API is not available, such as some development environments.
• force_path_style: Forces the use of path-style URLs when interacting with the Object Storage service. This is commonly used when working with S3-compatible storage services.

Migrate Existing State

Now that the configuration has been completed, we can proceed to migrate the state data. This process involves transferring the data from the current system to the new one in a secure and efficient manner. It is important to ensure that all data is accurately transferred and that there are no discrepancies or errors during the migration process.

Initialize Terraform

Start by running the following command to initialize Terraform with the new backend configuration:

terraform init
Initializing modules...

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value:

This command will prompt you to migrate your state. Confirm the migration by entering “yes” to proceed.

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of hashicorp/oci from the dependency lock file
- Using previously-installed hashicorp/oci v5.6.0

╷
│ Warning: Additional provider information from registry
│ 
│ The remote registry returned warnings for registry.terraform.io/hashicorp/oci:
│ - For users on Terraform 0.13 or greater, this provider has moved to oracle/oci. Please update
│ your source in required_providers.
╵

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

💡 Using the -migrate-state option in terraform init command can perform automatic migration of the state. This is useful if you already a different backend configured for your state.

Verify and Test

After the migration, you can verify that the state is now stored in OCI Object Storage by running terraform state list or finding the object in the blob storage in the console. Remember to Test your infrastructure deployments to ensure that everything is functioning as expected.

Conclusion

Migrating your Terraform state from a local file system to OCI Object Storage is a strategic move that enhances security, collaboration, scalability, and reliability in your infrastructure management process. By following the steps outlined in this guide, you can seamlessly transition to a more robust state management solution and take full advantage of Oracle Cloud Infrastructure's capabilities. Happy Terraforming!