Setting up Linode Object Storage as a Terraform backend

In this post, we'll go through the setup process for configuring Linode Object Storage as a remote backend for your terraform state. This enables you to have your state stored remotely, allowing you to safely store your terraform configuration publicly without worrying about leaking sensitive information through the state.

Whilst terraform doesn't directly support Linode as a backend, Linode Object Storage is S3 compatible which means that with only a few extra settings, we can use it as an S3 backend.

Creating the Bucket

Now we could just go into the Linode dashboard and create the bucket manually but we're going to use terraform so why not use that to create the bucket as well! This means that the state for the bucket will need to be committed to the repository but that should be fine as we don't need to store any sensitive information to create it.

In the folder where you want to create your terraform configuration, create a new folder to hold this initial terraform code and cd into it. I'll call mine init-state.

cd ~/src/infra/terraform
mkdir init-state
cd init-state

Now we need to add the terraform code to create the bucket. This is pretty simple but can be extended to add extra configuration such as lifecycle rules or versioning.

Note
At the time of writing this article, the current terraform version is 1.0.11. If you're using a newer version, it's possible that the syntax/format could have changed so make sure you're using the correct syntax for your version.

First off, we create our main.tf file, setup the terraform provider configuration, and register the provider. We'll be using the Linode provider. Make sure to use the latest version of the provider which at the time of writing is 1.25.0.

terraform {
  required_providers {
    linode = {
      source  = "linode/linode"
      version = "1.25.0"
    }
  }
}

provider "linode" {}

Now we can lookup the storage cluster you want to use, I'm using Frankfurt, DE as it's currently the only European region that supports object storage.

data "linode_object_storage_cluster" "primary" {
  id = "eu-central-1"
}

Finally, we can define our bucket and give it a name! The name must be unique across all Linode object storage, not just in your account.

resource "linode_object_storage_bucket" "tf_state" {
  cluster = data.linode_object_storage_cluster.primary.id
  label   = "my-tf-state"
}

All together, this is what we have:

terraform {
  required_providers {
    linode = {
      source  = "linode/linode"
      version = "1.25.0"
    }
  }
}

provider "linode" {}

data "linode_object_storage_cluster" "primary" {
  id = "eu-central-1"
}

resource "linode_object_storage_bucket" "tf_state" {
  cluster = data.linode_object_storage_cluster.primary.id
  label   = "my-tf-state"
}

There's one last thing we need to do before we can apply our config and create the bucket, and that is to generate a Linode API token. To do that, go to the API Tokens page and click the Create a Personal Access Token button in the top right. Give it a name, an expiry, and some permissions. You should limit the permissions to what you plan on managing with terraform. For the sake of this guide, I set everything to Read/Write.

Now click Create Token and copy the token that displays on the screen. Make sure you save the token somewhere safe like a password manager as once you close the popup, you won't be able to see the token again.

For the Linode provider to access this token, you can just run the terraform commands and it will prompt for the token. If you don't want to enter it each time, you can set the LINODE_TOKEN environment variable instead with the token as the value.

export LINODE_TOKEN=<paste-token-here>

We should now be ready to initalize terraform and apply the bucket configuration!

terraform init
terraform apply

This should download the Linode provider and then output a plan which looks like this:

Terraform will perform the following actions:

  # linode_object_storage_bucket.state will be created
  + resource "linode_object_storage_bucket" "state" {
      + acl          = "private"
      + cluster      = "eu-central-1"
      + cors_enabled = true
      + id           = (known after apply)
      + label        = "my-tf-state"
      + versioning   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

If you're happy with the output, go ahead and type yes to create the bucket!

Before you commit everything to your repo, make sure to add .terraform to your .gitignore as it contains ephemeral data like the downloaded Linode provider which shouldn't be stored in your version control system.

Configure Storage Access

For terraform to be able to use the bucket as a backend, it needs access keys to allow it to read and write to the bucket. To create these keys, go to the Access Keys page and click the Create Access Key button. In the panel that opens, give the key a label, enable limited access, and allow Read/Write access to your state bucket:

Click the Create Access Key button and you should see your keys. As with the API Token from the last section, once you dismiss the popup you can't access the keys again so make sure you have stored them somewhere safe!

Now to enable terraform to read these keys without having to store them in the code, you can either store them in environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or in the AWS shared credentials configuration. I will go over the latter as the former is the same process as the API token.

The AWS shared credentials configuration consists of two files, ~/.aws/config and ~/.aws/credentials. The first file defines a profile which holds some general configuration like the output format and the default region. The second file contains the keys.

Let's start with the profile. Create ~/.aws/config and define the linode-s3 profile. Having a separate profile means that if you have existing AWS profiles, this won't cause you any conflicts. You should set the region setting to the region you chose when you created your state bucket:

[profile linode-s3]
output = text
region = eu-central-1

Next we need to create the ~/.aws/credentials file. This should set the credentials for the linode-s3 profile. I would recommend restricting the permission on this file to 600 which only allows read/write access to the owner of the file.

[linode-s3]
aws_access_key_id=<access-key>
aws_secret_access_key=<secret-key>

With these files in place, we should be ready for the last step, configuring the terraform backend!

Configure Terraform

To configure the terraform backend, we should change to the directory where you want to store your terraform code and create a file named backend.tf. It isn't required to be in it's own file but it does help to keep your code nicely organised. This file should contain the following terraform block:

terraform {
  backend "s3" {
    endpoint                    = "eu-central-1.linodeobjects.com"
    profile                     = "linode-s3"
    skip_credentials_validation = true
    bucket                      = "my-tf-state"
    key                         = "infra/state.json"
    region                      = "eu-central-1"
  }
}

This defines an s3 backend and points it to the Linode object storage endpoint for your chosen region, which in this example is eu-central-1. Next we tell terraform to load the credentials from the linode-s3 profile we created in the AWS config files. It's important to set skip_credentials_validation to true as otherwise terraform will reach out to AWS STS to try to validate the access keys which will obviously fail. Finally we set the bucket name, key, and region. The key is the path in the bucket to the terraform state file.

The last thing to do is initialise terraform with the new backend. Once that's done, you should be ready to create some resources with terraform and be safe in the knowledge that your state file is stored in a private Linode bucket!

Note
Terraform will still store a local copy of the state on your machine to make it easier to work with. It is VERY important to make sure you add the .terraform directory to your .gitignore as otherwise this state file could be accidentally committed and all of our hard work will be for nothing!

terraform init

You can test it out by creating a Linode resource and running terraform apply:

terraform {
  required_providers {
    linode = {
      source  = "linode/linode"
      version = "1.25.0" # Current latest version as of 2021-12-03
    }
  }
}

provider "linode" {}

variable "root_password" {
  type        = string
  description = "Password for the root user"
  sensitive   = true
}

resource "linode_instance" "my_server" {
  label           = "my-server"
  image           = "linode/debian11"
  region          = "eu-west"
  type            = "g6-nanode-1"
}

Conclusion

You should now be set up and ready to start creating your terraform resources with the state being safe in your remote Linode bucket. Hopefully you've learned a little about the benefits of storing your state using a remote backend and are inspired to play around with some infrastructure as code!

It's worth noting that using Linode as a remote backend doesn't limit you to only managing Linode resources with your new setup. You can go ahead and use any of the official or community providers found on the terraform registry.

Have fun learning and playing with terraform!