Terraform is an open-source infrastructure as code software tool created by HashiCorp. Even though I am very familiar with it and have featured some of the really cool modules and content within my blog, I have never really used Terraform before. When I heard that the AWS-IA team had put together a new Managed Workflows for Apache Airflow (MWAA) module for Terraform, I knew the time had finally come to give this a go. I spent some time last week playing around with this, testing it and adding some tweaks to the docs. I thought I would use that time to take some notes and then put together this blog post.
So read on if you want to see how you can use Terraform to automate the configuration and deployment of your MWAA environments.
Installing Terraform
First of all, need to install it.
HashiCorp's documentation is great, and I was able to install Terraform via this page.
brew tap hashicorp/tap
brew install hashicorp/tap/terraform
which on my mac, produced output that looked like this
==> Tapping hashicorp/tap
Cloning into '/usr/local/Homebrew/Library/Taps/hashicorp/homebrew-tap'...
remote: Enumerating objects: 2210, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 2210 (delta 64), reused 69 (delta 60), pack-reused 2116
Receiving objects: 100% (2210/2210), 386.46 KiB | 752.00 KiB/s, done.
Resolving deltas: 100% (1373/1373), done.
Tapped 1 cask and 18 formulae (51 files, 540.4KB).
(base) @094459 ~ brew tap hashicorp/tapbrew install hashicorp/tap/terraform
(base) ✘ @094459 ~ brew install hashicorp/tap/terraform
==> Downloading https://releases.hashicorp.com/terraform/1.2.3/terraform_1.2.3_d
######################################################################## 100.0%
==> Installing terraform from hashicorp/tap
🍺 /usr/local/Cellar/terraform/1.2.3: 3 files, 67.4MB, built in 7 seconds
==> Running `brew cleanup terraform`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
terraform -version
Terraform v1.2.3
on darwin_amd64
As I tend to use Visual Studio Code, I also installed the extension that HashiCorp provide to work with Terraform files within Code.
Creating our MWAA Terraform configuration
The MWAA Terraform module can be found at https://github.com/aws-ia/terraform-aws-mwaa/tree/main. Clone the repo into your local workspace.
git clone https://github.com/aws-ia/terraform-aws-mwaa/tree/main
From the examples/basic folder, we have a simple MWAA stack that we can use in order to test that everything is working.
├── README.md
├── dags
│ └── hello_world_dag.py
├── main.tf
├── mwaa
│ └── requirements.txt
├── outputs.tf
├── providers.tf
└── variables.tf
The README.md contains a quickstart on how to deploy your first environment. This post will walk you through the steps, but I still suggest you check that out. There are a number of key files that you will need to understand before you deploying your first MWAA environment.
mwaa and dags folders
You will notice that we have a sample Apache Airflow DAG (hellow_world_dag.py) that we want to deploy as we build our MWAA environment. We also have a requirements.txt file that we will want to upload (it is currently empty).
This example shows you how you can do this as we will see a bit later on. For the moment, all you need to be aware of is that these are resources that you want to deploy as you build out your MWAA environment.
variables.tf
This file contains configuration options that you can alter to change your MWAA environment - the name of the environment, the AWS region and default tags. For this demo, these are the values I am using.
variable "name" {
description = "Name of MWAA Environment"
default = "terraform-mwaa"
type = string
}
variable "region" {
description = "region"
type = string
default = "eu-central-1"
}
variable "tags" {
description = "Default tags"
default = {"env": "test", "dept": "AWS Developer Relations"}
type = map(string)
}
variable "vpc_cidr" {
description = "VPC CIDR for MWAA"
type = string
default = "10.1.0.0/16"
}
main.tf
The main.tf contains the main Terraform configuration file that will deploy resources, using values that are contained in the variables.tf. Lets take a look at this (main.tf) in more detail.
At the top of the file we have the following, the important value here is "bucket_name" which will configure a unique S3 bucket which will be used by your MWAA environment. This is important as the subsequent uploading of the sample DAGs, the requirements.txt as well as all the IAM policy documents all use this value.
locals {
azs = slice(data.aws_availability_zones.available.names, 0, 2)
bucket_name = format("%s-%s", "aws-ia-mwaa", data.aws_caller_identity.current.account_id)
}
Next in the file, we have the section that creates and uploads the sample DAG and requirements files
#-----------------------------------------------------------
# Create an S3 bucket and upload sample DAG
#-----------------------------------------------------------
#tfsec:ignore:AWS017 tfsec:ignore:AWS002 tfsec:ignore:AWS077
resource "aws_s3_bucket" "this" {
bucket = local.bucket_name
tags = var.tags
}
resource "aws_s3_bucket_acl" "this" {
bucket = aws_s3_bucket.this.id
acl = "private"
}
resource "aws_s3_bucket_versioning" "this" {
bucket = aws_s3_bucket.this.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
bucket = aws_s3_bucket.this.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
resource "aws_s3_bucket_public_access_block" "this" {
bucket = aws_s3_bucket.this.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
# Upload DAGS
resource "aws_s3_object" "object1" {
for_each = fileset("dags/", "*")
bucket = aws_s3_bucket.this.id
key = "dags/${each.value}"
source = "dags/${each.value}"
etag = filemd5("dags/${each.value}")
}
# Upload plugins/requirements.txt
resource "aws_s3_object" "reqs" {
for_each = fileset("mwaa/", "*")
bucket = aws_s3_bucket.this.id
key = each.value
source = "mwaa/${each.value}"
etag = filemd5("mwaa/${each.value}")
}
In this section, we define the name of the MWAA environment we want to use, the version of Apache Airflow (1.12, 2.0.2, or 2.2.2), and the size of the MWAA Worker nodes (mw1.small, mw1.medium, or mw1.large). We then define the name of the dags folder which Apache Airflow will use as the "dags folder" to search for DAGs to run. Finally, you can optionally set a plugins.zip and requirements.zip file and location, but these are not set by default.
name = "basic-mwaa"
airflow_version = "2.2.2"
environment_class = "mw1.medium"
dag_s3_path = "dags"
#plugins_s3_path = "plugins.zip"
#requirements_s3_path = "requirements.txt"
If you want to set the plugins_s3_path or requirements_s3_path you will need to set these here and then configure/deploy the plugins.zip and requirements.txt separately.
The next section configures the logging for the various MWAA services. This section allows us to define the logging verbosity for the different MWAA services. Bear in mind that there is a cost associated with this, so understand this before configuring these. The values you can use are CRITICAL, ERROR, WARNING, INFO, or DEBUG.
logging_configuration = {
dag_processing_logs = {
enabled = true
log_level = "INFO"
}
scheduler_logs = {
enabled = true
log_level = "WARNING"
}
task_logs = {
enabled = true
log_level = "DEBUG"
}
webserver_logs = {
enabled = true
log_level = "INFO"
}
worker_logs = {
enabled = true
log_level = "INFO"
}
}
In the next section you can define some custom Apache Airflow configuration parameters if you need to do so. You can check out the MWAA documentation to find out more, but you might use these to tweak performance settings or enable AWS integration of things like AWS Secrets Manager.
airflow_configuration_options = {
"core.load_default_connections" = "false"
"core.load_examples" = "false"
"webserver.dag_default_view" = "tree"
"webserver.dag_orientation" = "TB"
}
Provides scaling settings for the Apache Airflow worker nodes, and then provides details of the VPC networking. You should probably not need to change these (network) settings.
min_workers = 1
max_workers = 25
vpc_id = module.vpc.vpc_id
private_subnet_ids = module.vpc.private_subnets
webserver_access_mode = "PUBLIC_ONLY"
source_cidr = ["10.1.0.0/16"]
Finally, these options allows you to define and bring your own AWS security groups, execution roles or S3 buckets to use within MWAA. If you do create your own, make sure that these meet the minimum requirements by checking the MWAA documentation. Also, you will need to comment out the sections above so that Terraform will configure the correct resources.
# create_security_group =
# source_bucket_arn =
# execution_role_arn =
Now that we have gone over these configuration, we are ready to try and deploy this. From my Visual Code IDE, I run the following command to initiate.
terraform init
which generates the following output
Initializing modules...
- mwaa in ../..
Downloading registry.terraform.io/terraform-aws-modules/vpc/aws 3.14.2 for vpc...
- vpc in .terraform/modules/vpc
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching ">= 3.63.0, ~> 4.20.0"...
- Installing hashicorp/aws v4.20.1...
- Installed hashicorp/aws v4.20.1 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
terraform plan
Displays what AWS resources Terraform will create. I will not display all the output as it will be different to yours but you can review it and you will see all the different resources that are about to be configured and deployed.
You are now ready to deploy your MWAA environment.
Deploying the MWAA environment
To deploy, you can now run "terraform apply", and having checked the output, answering "yes" when prompted if it looks all good.
terraform apply
The deployment will then begin. This is what my output looked like:
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
module.vpc.aws_vpc.this[0]: Creating...
module.vpc.aws_eip.nat[0]: Creating...
module.mwaa.aws_iam_role.mwaa[0]: Creating...
module.mwaa.aws_s3_bucket.mwaa[0]: Creating...
module.vpc.aws_eip.nat[0]: Creation complete after 1s [id=eipalloc-0b7a488bea6bfddfc]
module.mwaa.aws_iam_role.mwaa[0]: Creation complete after 2s [id=mwaa-executor20220628091608972100000001]
module.mwaa.aws_s3_bucket.mwaa[0]: Creation complete after 3s [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_bucket_versioning.mwaa[0]: Creating...
module.mwaa.aws_s3_bucket_acl.mwaa[0]: Creating...
module.mwaa.aws_s3_bucket_server_side_encryption_configuration.mwaa[0]: Creating...
module.mwaa.aws_s3_bucket_public_access_block.mwaa[0]: Creating...
module.mwaa.aws_s3_object.python_requirements[0]: Creating...
module.mwaa.aws_s3_object.plugins[0]: Creating...
module.mwaa.data.aws_iam_policy_document.mwaa: Reading...
module.mwaa.data.aws_iam_policy_document.mwaa: Read complete after 0s [id=3112667403]
module.mwaa.aws_iam_role_policy.mwaa[0]: Creating...
module.mwaa.aws_iam_role_policy.mwaa[0]: Creation complete after 0s [id=mwaa-executor20220628091608972100000001:mwaa-executor20220628091611718600000003]
module.mwaa.aws_s3_bucket_acl.mwaa[0]: Creation complete after 1s [id=mwaa-704533066374-20220628091608973700000002,private]
module.mwaa.aws_s3_bucket_public_access_block.mwaa[0]: Creation complete after 1s [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_bucket_server_side_encryption_configuration.mwaa[0]: Creation complete after 1s [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_object.plugins[0]: Creation complete after 1s [id=plugins.zip]
module.mwaa.aws_s3_object.python_requirements[0]: Creation complete after 1s [id=requirements.txt]
module.mwaa.aws_s3_bucket_versioning.mwaa[0]: Creation complete after 2s [id=mwaa-704533066374-20220628091608973700000002]
module.vpc.aws_vpc.this[0]: Still creating... [10s elapsed]
module.vpc.aws_vpc.this[0]: Creation complete after 12s [id=vpc-08c5b33e125c3e2df]
module.vpc.aws_subnet.public[1]: Creating...
module.vpc.aws_subnet.private[0]: Creating...
module.vpc.aws_subnet.private[1]: Creating...
module.vpc.aws_subnet.public[0]: Creating...
module.vpc.aws_route_table.public[0]: Creating...
module.mwaa.aws_security_group.mwaa[0]: Creating...
module.vpc.aws_internet_gateway.this[0]: Creating...
module.vpc.aws_route_table.private[0]: Creating...
module.vpc.aws_route_table.public[0]: Creation complete after 1s [id=rtb-026cb73aa7c373b6d]
module.vpc.aws_subnet.private[1]: Creation complete after 1s [id=subnet-02c22ce11ef5d1f24]
module.vpc.aws_route_table.private[0]: Creation complete after 1s [id=rtb-06a02df385b093898]
module.vpc.aws_internet_gateway.this[0]: Creation complete after 1s [id=igw-058e9b992d5def525]
module.vpc.aws_route.public_internet_gateway[0]: Creating...
module.vpc.aws_subnet.private[0]: Creation complete after 1s [id=subnet-014e17effc050d3f8]
module.vpc.aws_route_table_association.private[0]: Creating...
module.vpc.aws_route_table_association.private[1]: Creating...
module.vpc.aws_route_table_association.private[0]: Creation complete after 1s [id=rtbassoc-063cb70e81c051767]
module.vpc.aws_route_table_association.private[1]: Creation complete after 1s [id=rtbassoc-00ced4c22a1f470c0]
module.vpc.aws_route.public_internet_gateway[0]: Creation complete after 1s [id=r-rtb-026cb73aa7c373b6d1080289494]
module.mwaa.aws_security_group.mwaa[0]: Creation complete after 2s [id=sg-0fc61902a4cbbae48]
module.mwaa.aws_security_group_rule.mwaa_sg_outbound[0]: Creating...
module.mwaa.aws_security_group_rule.mwaa_sg_inbound[0]: Creating...
module.mwaa.aws_security_group_rule.mwaa_sg_inbound_vpn[0]: Creating...
module.mwaa.aws_mwaa_environment.mwaa: Creating...
module.mwaa.aws_security_group_rule.mwaa_sg_inbound[0]: Creation complete after 1s [id=sgrule-1253679849]
module.mwaa.aws_security_group_rule.mwaa_sg_outbound[0]: Creation complete after 1s [id=sgrule-2481519482]
module.mwaa.aws_security_group_rule.mwaa_sg_inbound_vpn[0]: Creation complete after 2s [id=sgrule-2489225287]
module.vpc.aws_subnet.public[0]: Still creating... [10s elapsed]
module.vpc.aws_subnet.public[1]: Still creating... [10s elapsed]
module.vpc.aws_subnet.public[0]: Creation complete after 11s [id=subnet-02c0205901abdb075]
module.vpc.aws_subnet.public[1]: Creation complete after 11s [id=subnet-06ad25fc78a2cf59f]
module.vpc.aws_route_table_association.public[1]: Creating...
module.vpc.aws_route_table_association.public[0]: Creating...
module.vpc.aws_nat_gateway.this[0]: Creating...
module.vpc.aws_route_table_association.public[1]: Creation complete after 1s [id=rtbassoc-0440eeba24fee7e0b]
module.vpc.aws_route_table_association.public[0]: Creation complete after 1s [id=rtbassoc-02c12d5499fc36282]
module.mwaa.aws_mwaa_environment.mwaa: Still creating... [10s elapsed]
module.vpc.aws_nat_gateway.this[0]: Still creating... [10s elapsed]
module.mwaa.aws_mwaa_environment.mwaa: Still creating... [20s elapsed]
...
After about 20-25 minutes, you should get the following output to show that the deployment has been completed
Apply complete! Resources: 30 added, 0 changed, 0 destroyed.
Outputs:
mwaa_arn = "arn:aws:airflow:eu-central-1:704533066374:environment/basic-mwaa"
mwaa_role_arn = "arn:aws:iam::704533066374:role/mwaa-executor20220628091608972100000001"
mwaa_security_group_id = "sg-0fc61902a4cbbae48"
mwaa_service_role_arn = "arn:aws:iam::704533066374:role/aws-service-role/airflow.amazonaws.com/AWSServiceRoleForAmazonMWAA"
mwaa_status = "AVAILABLE"
mwaa_webserver_url = "1008702f-c770-4b55-bfbf-8f9e6ee823c5.c9.eu-central-1.airflow.amazonaws.com"
We can now copy the "mwaa_webserver_url" into a browser, and then login using our AWS credentials to access our new MWAA environment.
As you can see from the screenshot, our sample DAG has also been uploaded into the environment and we can use this to test that our environment is working as expected.
Deleting our MWAA environment
So we have covered how to configure and deploy MWAA environments, now I will cover how you can clean up and remove them. The clean up process takes around 20 minutes to complete. There are some things to be aware of when deleting your MWAA resources.
First, the destroy process will clean up all resources, including the S3 bucket that contains your DAGs. If you need to keep these safe, make sure that you copy/move them to another location before cleaning up your MWAA environment.
Second, the CloudWatch log groups that are created when the MWAA environment is configured will also not be deleted. If you want to completely clean up your environment, remember to go to CloudWatch and search under log groups and delete as needed.
With that out of the way, to remove this new environment we created and clean up all the resources, we issue the "terraform destroy" command, and when prompted respond appropriately.
terraform destroy
this will generate a lot of output, displaying what resources will be removed. Here is a small portion of what my output looked like
module.mwaa.data.aws_caller_identity.current: Reading...
module.mwaa.data.aws_iam_policy_document.mwaa_assume: Reading...
module.mwaa.data.aws_region.current: Reading...
data.aws_availability_zones.available: Reading...
module.mwaa.data.aws_partition.current: Reading...
module.mwaa.data.aws_partition.current: Read complete after 0s [id=aws]
module.vpc.aws_vpc.this[0]: Refreshing state... [id=vpc-08c5b33e125c3e2df]
module.mwaa.data.aws_region.current: Read complete after 0s [id=eu-central-1]
module.mwaa.data.aws_iam_policy_document.mwaa_assume: Read complete after 0s [id=2236429369]
module.mwaa.aws_iam_role.mwaa[0]: Refreshing state... [id=mwaa-executor20220628091608972100000001]
data.aws_availability_zones.available: Read complete after 0s [id=eu-central-1]
module.vpc.aws_eip.nat[0]: Refreshing state... [id=eipalloc-0b7a488bea6bfddfc]
module.mwaa.data.aws_caller_identity.current: Read complete after 1s [id=704533066374]
module.mwaa.aws_s3_bucket.mwaa[0]: Refreshing state... [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_security_group.mwaa[0]: Refreshing state... [id=sg-0fc61902a4cbbae48]
module.vpc.aws_route_table.private[0]: Refreshing state... [id=rtb-06a02df385b093898]
module.vpc.aws_route_table.public[0]: Refreshing state... [id=rtb-026cb73aa7c373b6d]
module.vpc.aws_subnet.private[0]: Refreshing state... [id=subnet-014e17effc050d3f8]
module.vpc.aws_internet_gateway.this[0]: Refreshing state... [id=igw-058e9b992d5def525]
module.vpc.aws_subnet.private[1]: Refreshing state... [id=subnet-02c22ce11ef5d1f24]
module.vpc.aws_subnet.public[0]: Refreshing state... [id=subnet-02c0205901abdb075]
module.vpc.aws_subnet.public[1]: Refreshing state... [id=subnet-06ad25fc78a2cf59f]
module.mwaa.aws_security_group_rule.mwaa_sg_outbound[0]: Refreshing state... [id=sgrule-2481519482]
module.mwaa.aws_security_group_rule.mwaa_sg_inbound[0]: Refreshing state... [id=sgrule-1253679849]
module.mwaa.aws_security_group_rule.mwaa_sg_inbound_vpn[0]: Refreshing state... [id=sgrule-2489225287]
module.vpc.aws_route.public_internet_gateway[0]: Refreshing state... [id=r-rtb-026cb73aa7c373b6d1080289494]
module.vpc.aws_route_table_association.private[0]: Refreshing state... [id=rtbassoc-063cb70e81c051767]
module.vpc.aws_route_table_association.private[1]: Refreshing state... [id=rtbassoc-00ced4c22a1f470c0]
module.vpc.aws_nat_gateway.this[0]: Refreshing state... [id=nat-0d73ce39bbe46fdd9]
module.vpc.aws_route_table_association.public[0]: Refreshing state... [id=rtbassoc-02c12d5499fc36282]
module.vpc.aws_route_table_association.public[1]: Refreshing state... [id=rtbassoc-0440eeba24fee7e0b]
module.vpc.aws_route.private_nat_gateway[0]: Refreshing state... [id=r-rtb-06a02df385b0938981080289494]
module.mwaa.aws_s3_bucket_public_access_block.mwaa[0]: Refreshing state... [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_bucket_versioning.mwaa[0]: Refreshing state... [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_object.plugins[0]: Refreshing state... [id=plugins.zip]
module.mwaa.aws_s3_object.python_requirements[0]: Refreshing state... [id=requirements.txt]
module.mwaa.aws_s3_bucket_server_side_encryption_configuration.mwaa[0]: Refreshing state... [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_bucket_acl.mwaa[0]: Refreshing state... [id=mwaa-704533066374-20220628091608973700000002,private]
module.mwaa.aws_mwaa_environment.mwaa: Refreshing state... [id=basic-mwaa]
module.mwaa.data.aws_iam_policy_document.mwaa: Reading...
module.mwaa.data.aws_iam_policy_document.mwaa: Read complete after 0s [id=3112667403]
module.mwaa.aws_iam_role_policy.mwaa[0]: Refreshing state... [id=mwaa-executor20220628091608972100000001:mwaa-executor20220628091611718600000003]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
# module.mwaa.aws_iam_role.mwaa[0] will be destroyed
- resource "aws_iam_role" "mwaa" {
- arn = "arn:aws:iam::704533066374:role/mwaa-executor20220628091608972100000001" -> null
- assume_role_policy = jsonencode(
...
...
You will be prompted to enter "yes" to confirm you want to delete the MWAA environment. It will then start cleaning up the resources, and then display the following:
module.mwaa.aws_mwaa_environment.mwaa: Still destroying... [id=terraform-mwaa, 2m40s elapsed]
module.mwaa.aws_mwaa_environment.mwaa: Still destroying... [id=terraform-mwaa, 2m50s elapsed]
module.mwaa.aws_mwaa_environment.mwaa: Still destroying... [id=terraform-mwaa, 3m0s elapsed]
module.mwaa.aws_mwaa_environment.mwaa: Still destroying... [id=terraform-mwaa, 3m10s elapsed]
module.mwaa.aws_mwaa_environment.mwaa: Still destroying... [id=terraform-mwaa, 3m20s elapsed]
as it cleans up the MWAA environment. This will take approx. 20 minutes.
Feedback
I plan to build upon this blog post and revisit and provide Terraform build files for some of the other MWAA related articles that I have done that show MWAA with other AWS resources such as Amazon EMR, Amazon Athena, Amazon RedShift and more.
All the resources you need are available at the Terraform module page, https://github.com/aws-ia/terraform-aws-mwaa/tree/main and make sure you check out the examples which is what I have used in this post.
The MWAA Terraform module is also available in the Terraform registry.
The AWS that have put this new Terraform module together would love your feedback. Does this work as you expect it? What examples would you like included? Did you try this and find any errors or quirks? Please let us know, either directly by raising an issue, or via comments below.
Many thanks.
Oldest comments (0)