Cross-Account Deployment with AWS SAM

AWS SAM is a serverless framework that allows you to easily develop and deploy serverless applications in AWS environments.
Today, I will describe a couple of methods to execute cross-account SAM deployments that I have recently come across.
When we talk about multi-account deployment, we typically mean that your team has a central AWS account for devops-related activities, such as all of your CI/CD (Continuous Integration and Continuous Deployment) pipelines, which are in charge of deploying infrastructure and code changes to each application environment as the pull requests are merged.

Problem

There is a SAM project in your codebase that has some lambda functions to run. Requirement is to deploy this SAM project in the application account (lets call it Dev-Account) but via the devops account (lets call it DevOps-Account)

Resolution

In Dev-Account, deploy below resources:

• We need to use a cross-account IAM role in the Dev-Account that will be assumed by DevOps-Account for carrying out the needed deployment actions.
  • Put all permissions that DevOps-Account needs to execute the deployment in Dev-Account.
  • Put the DevOps-Account in the trust-relationship of this role.

(Trust entity would look like below)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::{DevOpsAccountID}:root"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

In DevOps-Account, deploy below resources:

• An AWS CodePipeline pipeline which has at-least 2 stages. Let's call them Source and Deploy. You can add more stages as per your need) (https://docs.aws.amazon.com/codepipeline/latest/userguide/pipelines-create.html)
• An S3 bucket where CodePipeline stores the artifacts coming from different stages/actions of the pipeline. (You can place this bucket either in Dev-Account or DevOps-Account, just update the policy accordingly)
• A Customer-Managed KMS key in the DevOps-Account that is used by CodePipeline to encrypt/decrypt the objects in the artifacts bucket.
• The Deploy Action in the CodePipeline must invoke a CodeBuild project.(read more about it in below section) This project is essentially responsible to building and deploying our SAM application. (sam build, sam deploy commands).

Each CodeBuild build is supposed to obtain the buildspec file from the pipeline stage input artifacts.

CodeBuild Project Location

Now, we need to decide about the location of the CodeBuild project

Integration tools, such as Codebuild, are typically stored in the devops/operations account alongside pipelines, but they can be moved to your application account as well. This is where the solution design can take several forms, and you must decide which one is best for you. 

1. CodeBuild project placed in Dev-Account

I designed this solution myself, without online assistance like AWS forums or other platforms. Maybe you have already seen things like this, but it works.

• Spin-up a CodeBuild project in the Dev-Account.
• Create an IAM service role for CodeBuild in the Dev-Account. Provide the necessary permissions to this role, like CodePipeline artifacts bucket permissions, DevOps-Account KMS key permissions, CloudFormation permissions for SAM, and others as required.

(Find sample policy below)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:DescribeKey",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*"
            ],
            "Resource": [
                "{DevOpsAccountKMSKeyArn}",
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "cloudformation:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:Put*"
                "s3:ListBucket",
            ],
            "Resource": "{ArtifactsBucketArn}/*"
        }
    ]
}

Ensure that the artifacts bucket policy and DevOps-Account KMS Key Policy have access permissions for the Dev-Account CodeBuild service role.

• Create an IAM service role for CodePipeline in DevOps-Account that has CodePipeline service in the trust-relationship and sts:AssumeRole permission to assume the cross-account role in Dev-Account.
  • It must have permission to invoke the CodeBuild project in the Dev-Account.

(Find the sample policy below)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": [
                "{CrossAccountRoleArn in DevAccount}"
            ],
            "Effect": "Allow",
            "Sid": "CrossAccountAssumeRolePolicy"
        },
        {
            "Action": [
                "codestar-connections:UseConnection"
            ],
            "Resource": "*",
            "Effect": "Allow",
            "Sid": "AllowCodeStarConnection"
        },
        {
            "Action": [
                "codebuild:StartBuild",
                "codebuild:BatchGetBuilds",
                "codebuild:BatchGetBuildBatches",
                "codebuild:StartBuildBatch"
            ],
            "Resource": [
                "arn:aws:codebuild:{Region}:{DevAccountId}:project/*"
            ],
            "Effect": "Allow",
            "Sid": "CodeBuildPolicy"
        },
        {
            "Action": [
                "kms:DescribeKey",
                "kms:GetKeyPolicy",
                "kms:List*",
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:Generate*"
            ],
            "Resource": "{CustomerManagedKeyArn in DevOps Account}",
            "Effect": "Allow",
            "Sid": "KMSPolicy"
        }
        {
            "Action": [
                "s3:*"
            ],
            "Resource": "*",
            "Effect": "Allow",
            "Sid": "S3Policy"
        },
        {
            "Action": [
                "iam:PassRole"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow",
            "Sid": "PassRolePolicy"
        }
    ]
}

• Assign the Dev-Account cross-account IAM role to the Deploy stage action of the pipeline.

2. CodeBuild project placed in DevOps-Account

I guess this is the most effective approach, at-least for me :) because I want all my CD/CD resources to be in one place. It makes the management and tracking of resources easier.

• Create the CodeBuild project in the DevOps-Account.

No need to assign any IAM role to the Deploy stage action in the pipeline because the CodeBuild project is in the DevOps-Account and the action will inherit the CodeBuild permissions from the parent pipeline service role.

• Create an IAM service role for CodeBuild in the DevOps-Account that has CodeBuild service in the trust-relationship and sts:AssumeRole permission to assume the cross-account role in Dev-Account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": [
                "{CrossAccountRoleArn in DevAccount}"
            ],
            "Effect": "Allow",
            "Sid": "CrossAccountAssumeRolePolicy"
        },
        .
        .
        .
        .
        .

    ]
}

• Since, CodeBuild project is in the DevOps-Account, so SAM commands will deploy the resources in the DevOps-Account that is not correct. So, we need to set the Dev-Account credentials in the CodeBuild project in order for it to deploy the resources into the Dev-Account.
  • Use the AWS Security-Token-Service to get the temporary credentials for the Dev-Account.

(Update your buildspec file like below)

version: 0.2

phases:
  build:
    commands:
      - echo getting AWS credentials for "$CROSSACCOUNT_SAM_ROLE"
      - |
        CREDENTIAL=$(aws sts assume-role \
          --duration-seconds 900
          --role-arn "$CROSSACCOUNT_SAM_ROLE" \
          --role-session-name SAMSession \
          --output text \
          --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken,Expiration]')
      - export AWS_ACCESS_KEY_ID=$(echo $CREDENTIAL | awk '{print $1}')
      - export AWS_SECRET_ACCESS_KEY=$(echo $CREDENTIAL | awk '{print $2}')
      - export AWS_SESSION_TOKEN=$(echo $CREDENTIAL | awk '{print $3}')
      - export SESSION_EXPIRATION=$(echo $CREDENTIAL | awk '{print $4}')
- sam build      
- sam deploy --no-confirm-changeset --no-fail-on-empty-changeset

$CROSSACCOUNT_SAM_ROLE environment variable in above buildspec file refers to the cross-account Dev-Account IAM role.

• Make sure that, cross-account IAM role has all necessary permissions needed to deploy the SAM resources.

SAM --role-arn arg

You might think that, there must be a way for the SAM to assume the cross-account IAM role in Dev-Account instead of generating the credentials explicitly. It would be a more cool🧊 solution, ain't it?

I tried that myself, there's a parameter in the SAM deploy command named --role-arn which basically CloudFormation service assumes to deploy the resources/changes.

(My buildspec was simple as below)

version: 0.2

phases:
  build:
    commands:
      - sam build
      - sam deploy --no-confirm-changeset --no-fail-on-empty-changeset --role-arn {CrossAccountDevRoleArn}

This results into below error

Cross-account pass role is not allowed (Service: AmazonCloudFormation; Status Code: 403; Error Code: AccessDenied; Request ID: d880bdd7-fe3f-11e7-8a8c-7dcffeae19ae)

Here, CodeBuild in DevOps-Account tries to pass a cross-account IAM role to the CloudFormation service in the Dev-Account and this is not possible.

You cannot use the PassRole permission to pass a cross-account role. read here

So, this hack wouldn't work🙂 and generating the cross-account credentials explicitly is the only solution as per my knowledge or that I could find online.

happy learning🚀

ref -> 1, 2, 3