Recently I have discovered a pattern that has emerged to become extremely handy when wanting to build out resources that are composed together, typically a one-to-many relationship.
One-to-Many
Now in terraform/cloud land the above image one can express the above image with something as simple as a security-group. The security group can be expressed as the "parent" and a security-group rule can would be expressed as a child of security-group. Thus, the security group is a composition of many security-group rules.
Foo Bar Code
locals {
map = {
"foo" = ["child1", "child4"]
"bar" = ["child2", "child5"]
"baz" = ["child3", "child6"]
}
out = merge([for parentKey, parentValue in local.map : {
for child in parentValue : "${parentKey}/${child}" => {
value = child
}
}
]...)
}
output "out" {
value = local.out
}
The above code loops through each key-value pair in the map and then it loops through each item in the array (the value of the respective pair).
The for expression in the array will create a map based on the key and item . This is responsible for grouping the maps by the keys of the initial map.
The next for expression in the map is responsible for the creation of the key - values based on the respective value (array of strings ). The key will get namespaced by the respective key (e.g. "foo") and the respective value of the element in the array (e.g. "child1").
Resulting in the composed key : "foo/child1"
The result of the two for expressions before the merge looks like such:
array = [
{
"foo/child1" = {
value = "child1"
},
"foo/child4" = {
value = "child4"
}
},
{
"bar/child2" = {
value = "child2"
},
"bar/child5" = {
value = "child5"
}
},
{
"baz/child3" = {
value = "child3"
},
"baz/child6" = {
value = "child6"
}
},
]
The ellipsis at the end of the array is similar to the spread operator in Javascript , similarly in golang it's a vararg .
What it does is spread the three maps as args in the merge function like such merge({...},{...},{...}).
From there the merge function handles the rest by merge the 3 maps into a single map. Just be careful with this as you need unique keys , otherwise the duplicated key will get overridden with the last respective value.
console output
+ out = {
+ "bar/child2" = {
+ value = "child2"
}
+ "bar/child5" = {
+ value = "child5"
}
+ "baz/child3" = {
+ value = "child3"
}
+ "baz/child6" = {
+ value = "child6"
}
+ "foo/child1" = {
+ value = "child1"
}
+ "foo/child4" = {
+ value = "child4"
}
}
Example
So obviously the above pattern is pretty nifty, let's find an example of how this pattern can become useful.
Lets say , you need to create a bunch of vpc-endpoints and you want each endpoint to have its own security-group because you need to control what can access these endpoints.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0"
}
}
}
locals {
## variables
default_vpc_id = "vpc-id"
subnet_ids = ["subnet-1-id", "subnet-2-id"]
endpoint_services = {
"s3" = {
service_name = "com.amazonaws.eu-west-1.s3",
private_dns_enabled = true,
vpce_name_tag = "s3-interface",
security_group_ingress_rules = {
"team-z-vpc-443" = {
port = 443,
cidr_blocks = ["10.0.0.0/24"],
protocol = "tcp",
description = "allow https vpc traffic in ",
},
"onprem-443" = {
port = 443,
cidr_blocks = ["192.168.0.0/16"],
protocol = "tcp",
description = "allow https onprem traffic in ",
},
},
},
"ddb" = {
service_name = "com.amazonaws.eu-west-1.dynamodb",
private_dns_enabled = true,
vpce_name_tag = "ddb-interface",
security_group_ingress_rules = {
"team-a-vpc-443" = {
port = 443,
cidr_blocks = ["10.0.1.0/24"],
protocol = "tcp",
description = "allow https vpc traffic in ",
},
},
}
}
## nested for loop for sg-rules
## not that sg_key_ref is used to reference the sg which is created based on the key-values of local.endpoint_services
endpoint_sg_ingress_rules = merge([for k, v in local.endpoint_services : {
for ingressKey, ingressValue in v.security_group_ingress_rules : "${k}/${ingressKey}" => merge({ sg_key_ref = k }, ingressValue)
}]...)
}
resource "aws_vpc_endpoint" "endpoint_services" {
for_each = local.endpoint_services
vpc_id = local.default_vpc_id
subnet_ids = local.subnet_ids
service_name = each.value.service_name
vpc_endpoint_type = "Interface"
security_group_ids = [
aws_security_group.this[each.key].id,
]
private_dns_enabled = each.value.private_dns_enabled
tags = {
Name = each.value.vpce_name_tag
}
}
resource "aws_security_group" "this" {
for_each = local.endpoint_services
vpc_id = local.default_vpc_id
name = "${each.value.vpce_name_tag}-sg"
description = "sg for endpoint service: ${each.value.service_name}"
tags = {
Name = "${each.value.vpce_name_tag}-sg"
}
}
resource "aws_security_group_rule" "ingress" {
for_each = local.endpoint_sg_ingress_rules
type = "ingress"
from_port = each.value.port
to_port = each.value.port
protocol = each.value.protocol
cidr_blocks = each.value.cidr_blocks
## referencing the parent ( security group) from which the rule is composed into
security_group_id = aws_security_group.this[each.value.sg_key_ref].id
}
The cool thing is: when you need to create another endpoint , all you do is add another object into the map that meets the interface thats defined by the variable(yes I didn't create a variable for this example).
Alternatives/Anti-pattern
I think the alternative to this pattern is to rather create a module for the logic that needs to composed and a for_each on the module by passing the respective properties/input variables into the module. The module would be responsible for handling the logic.
module "endpoints" {
for_each = local.endpoint_services
source = "../"
default_vpc_id = "vpc-id"
subnet_ids = ["subnet-1-id", "subnet-2-id"]
service_name = each.value.service_name
private_dns_enabled = each.value.private_dns_enabled
vpce_name_tag = each.value.vpce_name_tag
security_group_ingress_rules = each.value.security_group_ingress_rules
}
Thanks for taking the time read this article , hopefully it will become useful for you. Lastly, comment below how you would implement this sort of pattern !
Top comments (2)
We used to use this pattern extensively before modules supported
for_each. I'm interested to hear why you think module looping is an anti-pattern.We have found that using "local" modules as containers for this purpose has significantly improved code maintainability, at the cost of having to write a bit more "boilerplate" code in the beginning.
Nice writeup btw!
I appreciate it !😆
So I'd imagine that modules will be anti pattern to this because tf now supports for_each , so each iteration of the module will be "namespaced" by the key. So that would take away the need to have to have that sort prefix "s3/" for example because your module would compose/encapsulate all of the respective resources that we did the nested for loop logic with inside the module.
When you refer to local modules, do refer to modularising code that needs to be done on an interactive basis ? If so I could imagine the benefits! I've also noticed that even sometimes when there are a ton of resources that are related to a particular deployment , it's better to modularise it because reading through a million resources in a workdir is a pain when that workspace grows😂 perhaps that single instance of a module may become useful in the future if it's abstracted enough from implementation details