Terraform workflow can be ran using several methods. One of them is running Terraform workflow inside CI/CD pipeline. Running Terraform workflow inside CI/CD pipeline can have several benefits, such as: automate creation or provision resources, simplify collaboration between engineers/developers, etc.
Table of Contents
Scenario
In this blog, I will explain how to run Terraform workflow inside CI/CD pipeline which in this blog is Amazon CodeCatalyst. GitHub is used as source code repository and has been connected with CodeCatalyst. Next in CodeCatalyst pipeline/workflow, We will provision resources via Terraform. In this blog, Terraform is using S3 backend. Here is the topology used in this blog:
This blog is using reference from tutorial “Bootstrapping your Terraform automation with Amazon CodeCatalyst” written by @Cobus Bernard with some adjustment in configuration.
If You want to know more details about CodeCatalyst, please check CodeCatalyst documentation.
Prerequisites
- Before starting configuration, if you have not got AWS Builder ID account, please sign up first
- After sign up, go to Amazon CodeCatalyst web console to start project and configuration. Apply initial configuration that needed by CodeCatalyst such as:
- Creating space
- Connect CodeCatalyst space with AWS account
- Create and connect IAM role that will be used for running CodeCatalyst workflow/pipeline
- Connect GitHub source code repository
- Define environment that will be used by workflow/pipeline
- Please check my blog “Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst” to find reference for configuration above. Please check section
Amazon CodeCatalyst Pipeline/Workflow Configuration
- Prepare S3 bucket and DynamoDB lock table (optional) that will be used by Terraform backend
Configuration Steps
Directory Structure
Below is directory structure used in this blog:
.
├── .codecatalyst/
│ └── workflows/
│ └── tf-sbx2-vpc-apse1.yml
└── sandbox2/
├── vpc/
│ └── ap-southeast-1/
│ ├── main.tf
│ ├── resources.tf
│ └── variables.tf
└── tf-backend/
├── main.tf
├── resources.tf
└── variables.tf
Terraform Backend Configurations
Configuration start with provisioning resources that will be needed by Terraform backend such as S3 bucket and DynamoDB table. Beside that, We also need to prepare IAM role that will be needed to provision resources from CodeCatalyst workflow/pipeline.
-
At first, We haven’t had any resource for storing Terraform state in S3, so that we need to run and store terraform state on our local, for example: laptop. Later, We will move that state from local to S3. We start by defining configuration in sandbox2/tf-backend/main.tf
terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 5.11.0" } } required_version = ">= 1.3.0" } provider "aws" { region = var.aws_region }
-
Define variable region in file sandbox2/tf-backend/variables.tf
variable "aws_region" { type = string description = "AWS Region" default = "ap-southeast-1" }
-
Define resources S3, DynamoDB and IAM that needed by CodeCatalyst in file sandbox2/tf-backend/resources.tf. In this blog, IAM role CodeCatalyst has administrator access just for demo purposes. Please bear in mind to always use least privilege method on your production to prevent any security breaches
###### # S3 # ###### resource "aws_s3_bucket" "my_sandbox2_tfbucket" { bucket = "my-sandbox2-tfbucket" lifecycle { prevent_destroy = true } } resource "aws_s3_bucket_versioning" "my_sandbox2_tfbucket" { bucket = aws_s3_bucket.my_sandbox2_tfbucket.id versioning_configuration { status = "Enabled" } } resource "aws_s3_bucket_server_side_encryption_configuration" "my_sandbox2_tfbucket" { bucket = aws_s3_bucket.my_sandbox2_tfbucket.id rule { apply_server_side_encryption_by_default { sse_algorithm = "AES256" } } } resource "aws_s3_bucket_public_access_block" "my_sandbox2_tfbucket" { bucket = aws_s3_bucket.my_sandbox2_tfbucket.id block_public_acls = true block_public_policy = true ignore_public_acls = true restrict_public_buckets = true } ############ # DynamoDB # ############ resource "aws_dynamodb_table" "my_sandbox2_tflocks" { name = "my-sandbox2-tflocks" billing_mode = "PAY_PER_REQUEST" hash_key = "LockID" attribute { name = "LockID" type = "S" } } ####### # IAM # ####### data "aws_iam_policy_document" "codecatalyst_assume_role_policy" { statement { actions = ["sts:AssumeRole"] principals { type = "Service" identifiers = [ "codecatalyst.amazonaws.com", "codecatalyst-runner.amazonaws.com" ] } } } resource "aws_iam_role" "codecatalyst_admin" { name = "codecatalyst-admin" assume_role_policy = data.aws_iam_policy_document.codecatalyst_assume_role_policy.json } resource "aws_iam_role_policy_attachment" "codecatalyst_admin" { role = aws_iam_role.codecatalyst_admin.name policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess" }
Refer AWS keys that needed for running terraform workflow commands, for example by exporting environment variables as explained in this document
-
Run terraform cli commands that needed for provisioning resources
terraform init terraform plan terraform apply
Verify terraform apply is successful and resources successfully provisioned
-
Back to file sandbox2/tf-backend/main.tf, add additional configurations related to S3 backend
########################### # Terraform Configuration # ########################### terraform { backend "s3" { bucket = "my-sandbox2-tfbucket" key = "path/to/terraform.tfstate" region = "ap-southeast-1" dynamodb_table = "my-sandbox2-tflocks" encrypt = true } }
-
Run
terraform init
command again. Terraform will detect changes related to S3 backend and will move terraform state to S3 bucket that has been defined. Chooseyes
if asked for input
terraform init Initializing the backend... Do you want to copy existing state to the new backend? Pre-existing state was found while migrating the previous "local" backend to the newly configured "s3" backend. No existing state was found in the newly configured "s3" backend. Do you want to copy this state to the new "s3" backend? Enter "yes" to copy and "no" to start with an empty state. Enter a value: yes Releasing state lock. This may take a few moments... Successfully configured the backend "s3"! Terraform will automatically use this backend unless the backend configuration changes.
Go to Amazon Web Services (AWS) web console, go to S3 service menu. Verify
terraform.tfstate
file has been copied to defined S3 bucket/path
CodeCatalyst Workflow File Configurations
After configurations for terraform backend has been finished as described above, next we continue configure CodeCatalyst workflow/pipeline.
Before starting workflow configuration, please make sure to configure all configurations that needed by CodeCatalyst as mentioned in Prerequisites.
As mentioned in document Build, test, and deploy with workflows in CodeCatalyst, CodeCatalyst use workflow definition file as reference workflow/pipeline configuration and this file is saved in directory ~/.codecatalyst/workflows/
.
In this blog, I will create basic resources that needed by VPC on AWS. As mentioned on Directory Structure above, pipeline will be ran when there is changes inside directory sandbox2/vpc/ap-southeast-1
.
Below is full content of workflow file codecatalyst/workflows/tf-sbx2-vpc-apse1.yml
that used as example in this blog.
Name: tf-sbx2-vpc-apse1
SchemaVersion: "1.0"
Triggers:
- Type: PUSH
Branches:
- master
FilesChanged:
- sandbox2\/vpc\/ap-southeast-1\/.*
Actions:
terraform-plan:
Identifier: aws/build@v1
Inputs:
Sources:
- WorkflowSource
Environment:
Name: sandbox
Connections:
- Name: lanandra-sandbox
Role: codecatalyst-admin
Configuration:
Container:
Registry: DockerHub
Image: hashicorp/terraform:1.5.7
Steps:
- Run: cd sandbox2/vpc/ap-southeast-1
- Run: terraform fmt -check -no-color
- Run: terraform init -no-color
- Run: terraform validate -no-color
- Run: terraform plan -no-color -input=false
Compute:
Type: EC2
wait-period:
DependsOn:
- terraform-plan
Identifier: aws/build@v1
Inputs:
Sources:
- WorkflowSource
Environment:
Name: sandbox
Connections:
- Name: lanandra-sandbox
Role: codecatalyst-admin
Configuration:
Steps:
- Run: echo "Please wait for a while before terraform apply"
- Run: echo "If you wish to cancel terraform apply, please stop this run"
- Run: sleep 60
Compute:
Type: EC2
terraform-apply:
DependsOn:
- wait-period
Identifier: aws/build@v1
Inputs:
Sources:
- WorkflowSource
Environment:
Name: sandbox
Connections:
- Name: lanandra-sandbox
Role: codecatalyst-admin
Configuration:
Container:
Registry: DockerHub
Image: hashicorp/terraform:1.5.7
Steps:
- Run: cd sandbox2/vpc/ap-southeast-1
- Run: terraform init -no-color
- Run: terraform apply -auto-approve -no-color -input=false
Compute:
Type: EC2
Next, I will explain more detail each sections that defined on workflow file above.
-
Define workflow name. Then define how workflow will be ran. In this blog, workflow will be ran when there is a push to master branch but only ran when changes is under directory
sandbox2/vpc/ap-southeast-1/
(regular expression or regex is used to detect changes)
Triggers: - Type: PUSH Branches: - master FilesChanged: - sandbox2\/vpc\/ap-southeast-1\/.*
-
Define action. Action will be divided into 3 sections. The first one is used to run workflow
terraform plan
. In this section, action uses identifieraws/build@v1
. Then action will be ran on sandbox environment that has been connected with AWS account and also has IAM role associated with that account. InConfiguration
section, has been defined that action will be ran on top of container using specific version of terraform public image that pulled from DockerHub. Also has defined working directory and several terraform commands which includeterraform plan
. Lastly, define compute type which isEC2
Actions: terraform-plan: Identifier: aws/build@v1 Inputs: Sources: - WorkflowSource Environment: Name: sandbox Connections: - Name: lanandra-sandbox Role: codecatalyst-admin Configuration: Container: Registry: DockerHub Image: hashicorp/terraform:1.5.7 Steps: - Run: cd sandbox2/vpc/ap-southeast-1 - Run: terraform fmt -check -no-color - Run: terraform init -no-color - Run: terraform validate -no-color - Run: terraform plan -no-color -input=false Compute: Type: EC2
-
Next action that defined is called
wait-period
. This action is defined so that there is an interlude betweenterraform plan
andterraform apply
. In case there is something that unwanted in terraform plan, so that we have a time to not continue to next action or process which is terraform apply. This action is workaround because by the time this blog is released, there is no out of the box solution provided by CodeCatalyst, for example manual approval. So in case there is something unwanted, we can abort the workflow. In this action, I created a logic to pause the workflow for 60 seconds and this action is depends on previous action which isterraform plan
wait-period: DependsOn: - terraform-plan Identifier: aws/build@v1 Inputs: Sources: - WorkflowSource Environment: Name: sandbox Connections: - Name: lanandra-sandbox Role: codecatalyst-admin Configuration: Steps: - Run: echo "Please wait for a while before terraform apply" - Run: echo "If you wish to cancel terraform apply, please stop this run" - Run: sleep 60 Compute: Type: EC2
-
Next action that defined is action to run
terraform apply
. This action more or less similar with actionterraform-plan
, the difference is just terraform command that declared on configuration. In this section, action will run commands that needed forterraform apply
. This action is depends on previous action which iswait-period
terraform-apply: DependsOn: - wait-period Identifier: aws/build@v1 Inputs: Sources: - WorkflowSource Environment: Name: sandbox Connections: - Name: lanandra-sandbox Role: codecatalyst-admin Configuration: Container: Registry: DockerHub Image: hashicorp/terraform:1.5.7 Steps: - Run: cd sandbox2/vpc/ap-southeast-1 - Run: terraform init -no-color - Run: terraform apply -auto-approve -no-color -input=false Compute: Type: EC2
Example of Terraform Workflow Process in CodeCatalyst CI/CD
After configure CodeCatalyst workflow file, next I will create example of Terraform codes. In this blog, I will create VPC resources using public module terraform-aws-modules/vpc/aws as reference. Here is the initial configuration/codes that defined:
sandbox2/vpc/ap-souhteast-1/main.tf
###########################
# Terraform Configuration #
###########################
terraform {
backend "s3" {
bucket = "my-sandbox-tfbucket"
key = "path/to/terraform.tfstate"
region = "ap-southeast-1"
dynamodb_table = "my-sandbox-tflocks"
encrypt = true
}
}
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.11.0"
}
}
required_version = ">= 1.3.0"
}
provider "aws" {
region = var.aws_region
}
sandbox2/vpc/ap-souhteast-1/variables.tf
variable "aws_region" {
type = string
description = "AWS Region"
default = "ap-southeast-1"
}
variable "env_name" {
type = string
default = "sbx2"
}
sandbox2/vpc/ap-souhteast-1/resources.tf
#######
# VPC #
#######
data "aws_availability_zones" "available" {}
locals {
azs = slice(data.aws_availability_zones.available.names, 0, 3)
tags = {
"myTag:environment" = "sbx2"
"myTag:managedBy" = "terraform"
}
}
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "5.1.2"
name = var.env_name
cidr = "10.255.8.0/21"
azs = local.azs
private_subnets = ["10.255.8.0/24", "10.255.9.0/24", "10.255.10.0/24"]
public_subnets = ["10.255.15.0/24", "10.255.14.0/24", "10.255.13.0/24"]
enable_nat_gateway = false
single_nat_gateway = true
tags = local.tags
}
-
To conduct testing of Terraform workflow in CI/CD, I will change 1 line of code inside file sandbox2/vpc/ap-souhteast-1/resources.tf. I will activate NAT gateway with changing this line of code that current value is
false
enable_nat_gateway = true
-
As defined in workflow file, all changes inside directory sandbox/vpc/ap-southeast-1 will trigger workflow run. To verify, go to CodeCatalyst web console, go to space and project where codes resided. Then go to CI/CD menu, choose Workflows. Go to tab Runs, verify there is new run that currently running.
-
Go to run details by clicking name/id of run. Then We will be redirected to overview page of run. Click action name
terraform-plan
, on the right will be showed tray box that display details configuration/steps of action. For example We want to see details ofterraform plan
, go to tablog
, then click stepterraform plan -no-color -input-false
. Verify action succeeded -
Verify terraform plan show expected result. Terraform will create resources releated to NAT gateway as mentioned in point number 1
-
As defined in workflow file, after
terraform-plan
action ran successfully, next workflow will proceed next action which iswait-period
to give some interlude whether terraform plan will be executed to next action which isterraform-apply
or apply will be aborted. If plan has met expectation and We want to proceed toterraform apply
, no action needed, just wait 60 seconds and then verify action succeeded -
But, if want to abort workflow and don’t want continue to
terraform apply
, click stop button when action still in progress -
Let’s say We want to continue to
terraform apply
process, thenterraform-apply
action will be ran. We can see the details of the action inside tray box similar with previous actions. To verify resources has been provisioned viaterraform apply
, click stepterraform apply -auto-approve -no-color -input=false
-
Verify terraform apply has been expected and action is succeeded
-
Go to VPC service page in AWS web console to verify NAT gateway has been provisioned successfully
Summary
We have reached the last section of this blog. Here are some key takeaways that can be summarized:
- Amazon CodeCatalyst can act as alternative for CI/CD engine/tools that can be used to run Terraform workflow
- Amazon CodeCatalyst use IAM role to interact with AWS services. By using this method, Terraform doesn’t need to inject static credentials such as AWS Access Key and AWS Secret Key into the pipeline. This can’t help prevent security breaches
- Amazon CodeCatalyst can give seamless experience if We want to deploy application to AWS environments
- But some workflow or logic haven’t been provided by CodeCatalyst by the time this blog is released. Maybe there will be some improvement in the future
Please check out my GitHub repository to see source code example that used in this blog:
- Terraform backend configuration: tf backend
- Amazon CodeCatalyst workflow file: codecatalyst workflow
- Example of Terraform AWS VPC configuration: tf aws vpc
Please comment if you have any suggestions, critiques, or thoughts.
Hope this article will benefit you. Thank you!
Top comments (0)