Terraform: Write Configurations With Node.js

In this article, I'm going to share my experience in writing Terraform configurations using Node.js.

I'm going to showcase some benefits of writing Terraform configurations in Javascript/Typescript compared to writing native Terraform configurations.

Does Hashicorp recommend it?

Terraform also supports an alternative syntax that is JSON-compatible. This syntax is useful when generating portions of a configuration programmatically, since existing JSON libraries can be used to prepare the generated configuration files.

Above quote can be seen in Terraform documentation: https://www.terraform.io/docs/configuration/syntax-json.html.

Although Hashicorp doesn't really recommend using any tool to generate Terraform configurations, it does acknowledge that it is entirely possible and fine to generate Terraform configurations programmatically.

Tool

The tool that I use is called terraform-generator. It is available in npm registry: https://www.npmjs.com/package/terraform-generator.

What terraform-generator does is help generating Terraform configurations by utilising the capabilities of Node.js & Javascript/Typescript.
It currently supports generating configurations for Terraform 0.11 and 0.12.

The relationship between terraform-generator and Terraform configuration is similar to that of a query builder and database query, Typescript and Javascript or React.js and HTML & Web Javascript.

Syntax

The syntax is actually very similar to native Terraform syntax. Below is a comparison of creating a resource in native Terraform and in terraform-generator.

Terraform

resource "aws_vpc" "default" {
  cidr_block = "172.88.0.0/16"
}

terraform-generator

tfg.resource('aws_vpc', 'default', {
  cidr_block: '172.88.0.0/16'
});

Benefits

VSCode Extension

Last time I checked, there is no VSCode extension for Terraform 0.12. It is a pain when you want to navigate between resources, variables, etc.

By writing Javascript/Typescript in VSCode, you will have no problem doing that. It also provides all the usual benefits like auto-complete which is not available in native Terraform.

Shared Configurations

A Terraform project is a folder that contains one or many .tf files and a .terraform folder which contains the necessary Terraform plugins.

Let's say I have 3 projects for a system setup, they have a common provider, some common local variables and some common input variables, the common stuffs need to exist in all 3 projects. It causes my scripts to have duplicated elements and decreases the maintainability.

One remedy is to put the all shared configurations in another folder, outside of all project folders, then copy them (manually or via script, I will talk about the script that I wrote in next section) before executing the projects.

By using Node.js and terraform-generator, shared variables or code can be written in wherever you see fit, using them is just a matter of importing them.

Local Environmental States

Terraform will generate a terraform.tfstate and a terraform.tfstate.backup when we apply the project. One project can only have one state.

Let's say my project is going to be executed in 3 environments (development, staging & production), I won't be able to save the state in my local directory because I am going to have 3 different states, 1 for each environment. I will have to save the states in a remote storage (e.g. AWS S3).

One way to achieve saving states for multiple environments in local is to move the states to another folder, outside of the project folder after executing the project.

Below is an example of my Terraform folder structure and how I solve the shared configurations and environmental states problems with bash script. It increases complexity and decreases maintainability.

Folder structure

.
├── common                          # Shared configurations, to be copied to project folder before execution
|   ├── constants.tf
|   ├── env.tf
|   ├── provider.tf
|
├── env                             # Environmental variables, to be copied to project folder before execution
|   ├── dev.tfvars
|   ├── stg.tfvars
|   ├── prd.tfvars
|
├── outputs                         # Environmental states, to be copied to project folder before execution,
|   |                                 and then moved out from project folder after execution
|   ├── project1
|   |   ├── dev
|   |   |   ├── terraform.tfstate
|   |   |   ├── terraform.tfstate.backup
|   |   |
|   |   ├── stg
|   |   |   ├── ...
|   |   |
|   |   ├── prd
|   |       ├── ...
|   |
|   ├── project2
|   |   ├── ...
|   |
|   ├── project3
|       ├── ...
|
├── projects                        # Actual Terraform projects
|   ├── project1
|   |   ├── .terraform
|   |   ├── terraform.tf
|   |
|   ├── project2
|   |   ├── ...
|   |
|   ├── project3
|       ├── ...
|
├── run.sh                          # Bash script to do all the copying and moving of all the shared & environmental
                                      configurations and environmental states

run.sh

ACTION=$1
PROJECT=$2
ENV=$3

cd projects/$PROJECT

# Copy common tf, tfvars & tfstate to project folder
cp ../../common/* .
cp ../../env/$ENV.tfvars .
cp ../../outputs/$PROJECT/$ENV/* .

# Run terraform
terraform $ACTION -var-file=$ENV.tfvars

# Remove common tf & tfvars
rm -f constants.tf
rm -f env.tf
rm -f provider.tf
rm -f $ENV.tfvars

# Move tfstate to outputs folder
mkdir -p ../../outputs/$PROJECT/$ENV
mv terraform.tfstate ../../outputs/$PROJECT/$ENV
mv terraform.tfstate.backup ../../outputs/$PROJECT/$ENV

By using terraform-generator, while maintaining one source code, I will be able to generate multiple Terraform projects for multiple environments save local states in their respective project folder.

Below is an example of my terraform-generator folder structure to show you where generated Terraform configurations and states are located.

.
├── node_modules
|   ├── ...
|
├── outputs
|   ├── dev
|   |   ├── project1
|   |   |   ├── .terraform
|   |   |   ├── terraform.tf
|   |   |   ├── terraform.tfstate
|   |   |   ├── terraform.tfstate.backup
|   |   |
|   |   ├── project2
|   |   |   ├── ...
|   |   |
|   |   ├── project3
|   |       ├── ...
|   |
|   ├── stg
|   |   ├── ...
|   |
|   ├── prd
|       ├── ...
|
├── src
|   ├── constants
|   |   ├── ...
|   |
|   ├── env
|   |   ├── dev.env.ts
|   |   ├── index.ts
|   |   ├── stg.env.ts
|   |   ├── prd.env.ts
|   |
|   ├── projects
|       ├── project1
|       ├── ...
|       |
|       ├── project2
|       ├── ...
|       |
|       ├── project3
|       ├── ...
|
├── package.json
├── tsconfig.json

src folder contains my source code, it generates Terraform configuration to outputs folder according to the environment and project and the states is saved in the same folder as the generated Terraform configuration.

In short, I will have 3 similar Terraform configurations and 3 states while only maintaining 1 source code.

Variables

To use variables, Terraform requires us to write something like this:

variable "env" {
  type = string
}

variable "vpc_cidr" {
  type = string
}

You will have to remember to add a variable block whenever you introduce a new variable and remove the block whenever you decide to remove a variable to keep your configuration clean.

In Javascript, using a variable is just a matter of importing the variable or importing a JSON file.

If you are using Typescript and would like to declare an interface for all your variables, it is as simple as the following example:

export interface Variables {
  env: string;
  vpcCidr: string;
}

You can also make use of various libraries for your variable management, e.g. dotenv.

Conditionals

Terraform doesn't support if-else-statement, period.

By using Javascript/Typescript, you are free to using if-else or switch however you like.

The following example shows one of the use case of using if-else in my project:

const getAvailabilityZone = (idx: number): string => {
  const i = 3 % idx;
  if (i === 0) {
    return 'ap-southeast-1a';
  } else if (i === 1) {
    return 'ap-southeast-1b';
  } else {
    return 'ap-southeast-1c';
  }
};

for (let i = 0; i < 3; i++) {
  tfg.resource('aws_subnet', `subnet${i}`, {
    vpc_id: vpc.attr('id'),
    cidr_block: env.subnetCidrs[i],
    availability_zone: getAvailabilityZone(i)
  });
}

Without the for-loop (I will talk about it in next section) and if-else-statement, I will have to repeat the subnet creation 3 times to create them in 3 availability zones.

You can also use conditional to control resource attributes and resource creations, e.g.

if (env === 'production') {
  // create resource that is exclusive to production environment
}

tfg.resource('resource_type', 'resource_name', {
  attribute: env === 'production' ? 'some value': 'other value'
}

Loops

Terraform supports some sort of looping, e.g. count & for_each.

The limitation of Terraform's loop is that they are only supported by resource block but not module block as of now.

What if we want to create multiple resources in a loop? We will have to use count/for_each in each and every resource block. Wouldn't it be neater to have only 1 loop and create all the resources inside the loop?

Terraform's loop is one level looping (1 loop in the resource block). What if there is a need to have nested loop? E.g. using loop to create 3 security groups, for each security group, create 3 security group rules. Without nested loop, it is impossible to keep your configuration clean.

In Terraform, I will have to do something like this:

resource "aws_security_group" "sg" {
  count = 3
  ...
}

resource "aws_security_group_rule" "sgrule0" {
  count = length(aws_security_group.sg)
  security_group_id = aws_security_group.sg.*.id[count.index]
  ...
}

resource "aws_security_group_rule" "sgrule1" {
  count = length(aws_security_group.sg)
  security_group_id = aws_security_group.sg.*.id[count.index]
  ...
}

resource "aws_security_group_rule" "sgrule2" {
  count = length(aws_security_group.sg)
  security_group_id = aws_security_group.sg.*.id[count.index]
  ...
}

By using terraform-generator, you will be able do something like this:

for (let i = 0; i < 3; i++) {
  const sg = tfg.resource('aws_security_group', `sg${i}`, {
    ...
  });

  for (let j = 0; j < 3; j++) {
    tfg.resource('aws_security_group_rule', `sgrule${j}`, {
      security_group_id = sg.attr('id')
      ...
    });
  }
}

By using Javascript/Typescript, feel free to use any loop however you see fit. An example of using for-loop is shown in previous section.

Modules vs Functions

A Terraform module is similar to a Terraform project (project is also known as root module).

The process of creating and using a Terraform module is tedious, I will end up with another folder with another set of .tf files and the required plugins, and it doesn't even have direct access to my shared configurations and environmental variables.

For example, to create a module to simply create tags for all my resources, I will do the following:

variable "project_name" {
  type = string
}

variable "env" {
  type = string
}

output "tags" {
  value = {
    Project = var.project_name
    Env     = var.env
  }
}

In the project, I will do the following to use the module:

module "tags" {
  source        = "../../modules/tags"
  project_name  = local.projectName
  env           = var.env
}

resource "aws_vpc" "default" {
  cidr_block = var.vpc_cidr

  tags = merge(module.tags.tags, map(
    "Name", "vpc-${local.project_name}-${var.env}"
  ))
}

project_name and env are my common variables, they are shared by all projects under the same system setup, yet I still need to pass them into the module because it can't access them directly.

Besides, a module block has a set of fixed attributes and a fixed output, I can't pass in arguments and get back a tailored output, therefore I need to merge my variable tags with my constant tags manually. The process is tedious.

By using terraform-generator and Javascript's function, this is how I will do it:

const getTags = (name: string): Map => map({
  Name: `${name}-${constants.projectName}-${env.env}`,
  Project: constants.projectName,
  Env: env.env
});

tfg.resource('aws_vpc', 'default', {
  cidr_block: env.vpcCidr,
  tags: getTags('vpc')
});

It is obvious that the Typescript version is much easier and much more straightforward. It has access to my constants and environmental variables, it accepts arguments and return exactly what I need.

Other Capabilities

The strength of using Node.js to generate Terraform configurations is limitless, or should I say it is only limited by the capability which provided by Node.js and the Javascript world, which is much wider than what is provided by Terraform. You will be able to make use of any Node.js API and npm modules.

When We Should Use It

If you are a professional service provider, I can't advise you on whether using terraform-generator is a good move because it is not a widely adopted tool (yet). There are more considerations to think about, e.g. Will your customers accept the use of this unpopular tool? Is your company/colleagues open-minded enough to try it out? Will it have operational/maintenance issue in the future?

However if you are doing your own cloud infra setup and think that it might solve some of your pains in using Terraform, why not give it a try and tell me what you think in the comment section.