This tool is the result of a no so sad story as it should have been. We had a backup.
Until today, Terraform Cloud does not provide a mechanism to backup and restore the terraform states of the workspaces. This tool is the first version of a lambda that every time a workspace changes in the terraform state, the Terraform state will save in an S3 Bucket. So in case of disaster, the Terraform state can be pushed to the workspace.
Let us begin.
This tool is a serverless and event-driven solution. We are going to use:
Lambda repository
S3
API Gateway
Terraform Cloud Workspace Notifications
The workflow of the process is:
The terraform state change in terraform cloud because a new plan was applied or that a plan has been applied and ended in an errored state. In any case, the terraform state has changed so that we will save this new state.
This change of state triggers a notification in Terraform Cloud to a generic endpoint. This generic endpoint is an API Gateway endpoint that has a lambda function integrated.
The lambda function receives the payload from the terraform cloud notification, and this payload looks like this:
{
"resource": "/{proxy+}",
"path": "/hello/world",
"httpMethod": "POST",
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"cache-control": "no-cache",
"CloudFront-Forwarded-Proto": "https",
"CloudFront-Is-Desktop-Viewer": "true",
"CloudFront-Is-Mobile-Viewer": "false",
"CloudFront-Is-SmartTV-Viewer": "false",
"CloudFront-Is-Tablet-Viewer": "false",
"CloudFront-Viewer-Country": "US",
"Content-Type": "application/json",
"headerName": "headerValue",
"Host": "gy415nuibc.execute-api.us-east-1.amazonaws.com",
"Postman-Token": "9f583ef0-ed83-4a38-aef3-eb9ce3f7a57f",
"User-Agent": "PostmanRuntime/2.4.5",
"Via": "1.1 d98420743a69852491bbdea73f7680bd.cloudfront.net (CloudFront)",
"X-Amz-Cf-Id": "pn-PWIJc6thYnZm5P0NMgOUglL1DYtl0gdeJky8tqsg8iS_sgsKD1A==",
"X-Forwarded-For": "54.240.196.186, 54.182.214.83",
"X-Forwarded-Port": "443",
"X-Forwarded-Proto": "https"
},
"multiValueHeaders": {
"Accept": ["*/*"],
"Accept-Encoding": ["gzip, deflate"],
"cache-control": ["no-cache"],
"CloudFront-Forwarded-Proto": ["https"],
"CloudFront-Is-Desktop-Viewer": ["true"],
"CloudFront-Is-Mobile-Viewer": ["false"],
"CloudFront-Is-SmartTV-Viewer": ["false"],
"CloudFront-Is-Tablet-Viewer": ["false"],
"CloudFront-Viewer-Country": ["US"],
"Content-Type": ["application/json"],
"headerName": ["headerValue"],
"Host": ["gy415nuibc.execute-api.us-east-1.amazonaws.com"],
"Postman-Token": ["9f583ef0-ed83-4a38-aef3-eb9ce3f7a57f"],
"User-Agent": ["PostmanRuntime/2.4.5"],
"Via": ["1.1 d98420743a69852491bbdea73f7680bd.cloudfront.net (CloudFront)"],
"X-Amz-Cf-Id": ["pn-PWIJc6thYnZm5P0NMgOUglL1DYtl0gdeJky8tqsg8iS_sgsKD1A=="],
"X-Forwarded-For": ["54.240.196.186, 54.182.214.83"],
"X-Forwarded-Port": ["443"],
"X-Forwarded-Proto": ["https"]
},
"queryStringParameters": {
"name": "me"
},
"multiValueQueryStringParameters": {
"name": ["me"]
},
"pathParameters": {
"proxy": "hello/world"
},
"stageVariables": {
"stageVariableName": "stageVariableValue"
},
"requestContext": {
"accountId": "12345678912",
"resourceId": "roq9wj",
"stage": "testStage",
"domainName": "gy415nuibc.execute-api.us-east-2.amazonaws.com",
"domainPrefix": "y0ne18dixk",
"requestId": "deef4878-7910-11e6-8f14-25afc3e9ae33",
"protocol": "HTTP/1.1",
"identity": {
"cognitoIdentityPoolId": "theCognitoIdentityPoolId",
"accountId": "theAccountId",
"cognitoIdentityId": "theCognitoIdentityId",
"caller": "theCaller",
"apiKey": "theApiKey",
"apiKeyId": "theApiKeyId",
"accessKey": "ANEXAMPLEOFACCESSKEY",
"sourceIp": "192.168.196.186",
"cognitoAuthenticationType": "theCognitoAuthenticationType",
"cognitoAuthenticationProvider": "theCognitoAuthenticationProvider",
"userArn": "theUserArn",
"userAgent": "PostmanRuntime/2.4.5",
"user": "theUser"
},
"authorizer": {
"principalId": "admin",
"clientId": 1,
"clientName": "Exata"
},
"resourcePath": "/{proxy+}",
"httpMethod": "POST",
"requestTime": "15/May/2020:06:01:09 +0000",
"requestTimeEpoch": 1589522469693,
"apiId": "gy415nuibc"
},
"body": "{\n \"payload_version\":1,\n \"notification_configuration_id\":\"nc-vxvQ5SpJJ23oAWLk\",\n \"run_url\":\"https://app.terraform.io/app/mnsanfilippo/mnsanfilippo/runs/run-uyeKDPGKNpD9PxdC\",\n \"run_id\":\"run-uyeKDPGKNpD9PxdC\",\n \"run_message\":\"Queued manually using Terraform\",\n \"run_created_at\":\"2021-05-30T16:18:00.000Z\",\n \"run_created_by\":\"mnsanfilippo\",\n \"workspace_id\":\"ws-WzbRhM4ojZXjRvoP\",\n \"workspace_name\":\"mnsanfilippo\",\n \"organization_name\":\"mnsanfilippo\",\n \"notifications\":[\n {\n \"message\":\"Run Errored\",\n \"trigger\":\"run:errored\",\n \"run_status\":\"errored\",\n \"run_updated_at\":\"2021-05-30T16:19:14.000Z\",\n \"run_updated_by\":null\n }\n ]\n}"
}
The function is going to parse the body of the payload, searching for
- workspace_id
- workspace_name
- organization_name
- run_id
With that information, the function retrieves from Terraform Cloud the last tfstate of the workspace and will save it to S3.
To get all this working, this terraform module terraform-cloud-backup
will create and configure the resources needed.
module "example" {
source = "https://github.com/mnsanfilippo/terraform-cloud-backup.git?ref=main"
bucket_builds = var.bucket_builds
bucket_name = var.bucket_name
lambda_s3_key = var.lambda_s3_key
tf_token = var.tf_token
workspaces_ids = var.workspaces_ids
}
To get this up and running, we need a few stuff.
- Terraform Cloud Token.
- The build of the lambda in S3.
- The name of the S3 bucket where to save the tfstates.
- The list of the workspaces IDs to backup.
The variables should look like this:
Once that the module has been applied. There is a new notification in Terraform Cloud
And in S3 your tfstates are going to be saved as
bucket/organization/workspace/year/month/day/workspace-serial-stateID-runID.tfstate
This is all for today. In the following days, I will publish the Go-Tool to save all the tfstates on Terraform Cloud.
Top comments (0)