Validating CloudFormation Templates in AWS CodePipeline

CloudFormation Linter (cfn-lint) is a static code analysis tool that validates CloudFormation YAML and JSON templates against the CloudFormation Resource Specification. Engineers who work with CloudFormation normally run this tool locally on their machines to fix any issues with the templates and correct them immediately.

However... there are use cases where you work in bigger teams and need to manage hundreds of templates. Integrating cfn-lint in a CI/CD pipeline helps to enforce rules over shared CloudFormation templates, which in turn makes standardizing rules and guidelines over bigger teams easier. We are going to be integrating cfn-lint in AWS CodePipeline using a custom AWS CodeBuild project.

Overview

In this example, we have an AWS CodePipeline that contains the source code including the CloudFormation templates in an AWS CodeCommit repository. Next up we have the Validation stage where we have an AWS CodeBuild action that contains the cfn-lint tool. The templates will be scanned and the report will be sent to an encrypted S3 bucket. The last stage contains the actual deployment after the validation has finished.

Pipeline

If we're building the pipeline it looks as follows (snippet):

CodePipeline:
  Type: AWS::CodePipeline::Pipeline
  Properties:
    ArtifactStore:
      Location:
        Ref: ArtifactStore
      Type: S3
    Name: !Ref CloudFormationPipelineName
    RoleArn: !GetAtt CodePipelineRole.Arn
    RestartExecutionOnUpdate: False
    Stages:
    - Name: Source
      Actions:
      - Name: GetSource
        ActionTypeId:
          Category: Source
          Owner: AWS
          Version: 1
          Provider: CodeCommit
        OutputArtifacts:
        - Name: source
        Configuration:
          BranchName: master
          RepositoryName: cloudformation-codecommit-repo
    - Name: Validation
      Actions:
        - Name: Linter
          RunOrder: 1
          InputArtifacts:
            - Name: source
          ActionTypeId:
            Category: Build
            Owner: AWS
            Version: 1
            Provider: CodeBuild
          Configuration:
            ProjectName: !Ref CodeBuildLinter
    - Name: Deploy
      Actions:
      - Name: Dev
        RunOrder: 1
        InputArtifacts:
          - Name: source
        ActionTypeId:
          Category: Build
          Owner: AWS
          Version: 1
          Provider: CodeBuild
        Configuration:
          ProjectName: !Ref CodeBuildDev
ArtifactStore:
  Type: AWS::S3::Bucket
  Properties:
    BucketEncryption:
      ServerSideEncryptionConfiguration:
        - ServerSideEncryptionByDefault:
            SSEAlgorithm: aws:kms

The CloudFormation template snippet contains the three stages that were mentioned before Source -> Validation -> Deploy including an artifact bucket to save the artifacts from the build.

CodeBuild

The most important part of this pipeline is the validation stage which contains the CodeBuild step:

CodeBuildLinter:
    Type: AWS::CodeBuild::Project
    Properties:
      Artifacts:
        Type: CODEPIPELINE
      Description: Linting CloudFormation templates and sending reports to S3
      Environment:
        ComputeType: BUILD_GENERAL1_SMALL
        EnvironmentVariables:
          - Name: AWS_ACCOUNT_ID
            Value: !Sub ${AWS::AccountId}
          - Name: CFN_LINT_BUCKET
            Value: !Ref CfnLintBucket
          - Name: CFN_LINT_KMS_KEY_ARN
            Value: !Ref CfnLintKmsKeyId
          - Name: CODE_PIPELINE_NAME
            Value: !Ref ProvisioningPipelineName
        Image: aws/codebuild/standard:3.0
        Type: LINUX_CONTAINER
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Source:
        Type: CODEPIPELINE
        BuildSpec: buildspec-linter.yml
      TimeoutInMinutes: 20

The cfn-lint tool is added as a CodeBuild action in the validation stage of this pipeline. We add additional environment variables like the AWS Account ID, S3 report bucket, KMS Key and the pipeline name. The next step is to set up the buildspec of the CodeBuild project and add the container with the commands that need to be executed:

version: 0.2
phases:
  install:
    runtime-versions:
      python: 3.8
    commands:
      - apt-get update -y
      - apt-get install -y uuid-runtime
      - pip install cfn-lint
  build:
    commands:
      - |
        REPORT=$(uuidgen)-cfnlint-report.json
        set +e; cfn-lint -f json > $REPORT
        aws s3 cp $REPORT s3://$CFN_LINT_BUCKET/$AWS_ACCOUNT_ID/$CODE_PIPELINE_NAME/$(date +'%Y')/$(date +'%m')/$(date +'%d')/ --sse=aws:kms --sse-kms-key-id $CFN_LINT_KMS_KEY_ARN --acl bucket-owner-full-control

First, we install the required packages in the container. Next, we run cfn-lint and make sure the command won't exit (set +e) when it reports an error. In this situation, we don't want to stop exit CodeBuild if we discover an error because we use cfn-lint only for reporting.

In this setup, I've included a configuration file which specifies what templates need to be analyzed.

After the analysis, it will generate a report in JSON format which will be uploaded to an encrypted S3 bucket. As you can see the S3 CLI command is being used to transfer the report to the S3 bucket. We've included the variables which we passed from the AWS::CodeBuild::Project resource to dynamically create the object and enable KMS Encryption. After the report is uploaded to S3 it is ready for consumption for other applications.

KMS & S3 Bucket

If you enable AWS KMS, make sure to include the following policies for the CodeBuild role:

- Action:
    - s3:ListBucket
    - s3:ListBucketVersions
    - s3:ListObjects
    - s3:PutObject
    - s3:PutObjectAcl
  Effect: Allow
  Resource:
    - !GetAtt CfnLintBucket.Arn
    - !Sub ${CfnLintBucket.Arn}/*
  Sid: Allow S3 put CfnLintBucket
- Action:
    - kms:Encrypt
    - kms:ReEncrypt*
    - kms:GenerateDataKey*
    - kms:DescribeKey
    - kms:ListKeys
  Effect: Allow
  Resource: !GetAtt KMSKey.Arn
  Sid: Allow use of CMK in CfnLintBucket

The CodeBuild role is allowed to put the report in the destination bucket (CfnLintBucket). We also allow the role to encrypt the object using the KMSKey we created for the S3 Bucket. Then we need to explicitly allow the CodeBuildRole on the KMS Key policy, we can do that as follows:

- Sid: Allow encrypt use of the key by the CodeBuildRole
  Effect: Allow
  Principal:
    AWS:
      - !GetAtt CodeBuildRole.Arn
  Action:
    - kms:Encrypt
    - kms:ReEncrypt*
    - kms:GenerateDataKey*
    - kms:DescribeKey
    - kms:ListKeys
  Resource: '*'

To Summarize

We've set up a CodePipeline which adds cfn-lint as a validation stage to analyze your CloudFormation templates and send the reports to an encrypted S3 Bucket. This allows auditors for example to check how well the templates are built according to best practices.

👋 Enjoyed this article? Reach out in the comments below or on Twitter to let me know what you think of it.

If you found some value in reading this, please consider showing your support by sponsoring me. Thanks to your support, I'm able to continue doing what I enjoy the most, which is sharing my learnings with the Cloud Community.