Level 300
Table of Contents
In this post you can construct a cdk pipeline using Azure DevOps and based on first part of this series.
For this post the language for deployment will be .net, just to look from developer perspective for serveless deployment. Focus just in CDK application deployment rather than in the application indeed in other post the application will be the center.
Hands On
First review the requirements:
Requirements
- cdk >= 2.66.0
- AWS CLI >= 2.7.0
- Dotnet >= 6.0.405
- cdk-nag >=2.21.69
- checkov >= 2.1.229
- AWS Toolkit for Azure DevOps
- Azure DevOps Account
AWS Services and tools
- AWS Cloud Development Kit (CDK): is an open-source software development framework to define your cloud application resources using familiar programming languages.
- AWS Identity and Access Management (IAM): Securely manage identities and access to AWS services and resources.
- AWS IAM Identity Center (Successor to AWS Single Sign-On): helps you securely create or connect your workforce identities and manage their access centrally across AWS accounts and applications.
- AWS Key Management Service (AWS KMS): lets you create, manage, and control cryptographic keys across your applications and more than 100 AWS services.
AWS CloudFormation: Speed up cloud provisioning with infrastructure as code as code
AWS Security Token Service: web service that enables you to request temporary, limited-privilege credentials for AWS Identity and Access Management (IAM) users or for users you authenticate (federated users).
AWS Toolkit for Azure DevOps: adds tasks to easily enable build and release pipelines in Azure DevOps (formerly VSTS) and Azure DevOps Server (previously known as Team Foundation Server (TFS)) to work with AWS services including Amazon S3, AWS Elastic Beanstalk, AWS CodeDeploy, AWS Lambda, AWS CloudFormation, Amazon Simple Queue Service and Amazon Simple Notification Service, and run commands using the AWS Tools for Windows PowerShell module and the AWS CLI.
Step by step
First, the code follows a clean structure, for this example the main stack deploys a simple microservice compose for a SQS queue, a lambda and dynamodb table. The developer teams must use trunk-base development (trunkbaseddevelopment ). Figure 1 depicts the microservice architecture.
Figure 1. Simple microservice serverless architecture
Second, install .net Core Global Tool for Lambda and Amazon.Lambda.Template
dotnet tool install -g Amazon.Lambda.Tools
dotnet new -i Amazon.Lambda.Templates
Now, according to the best practices (https://aws.amazon.com/blogs/devops/best-practices-for-developing-cloud-applications-with-aws-cdk/ ) the application and infrastructure code lives in the same package.
The cdk project structure
First create a simple project a directory for your project and run cdk init command
.
mkdir read_message
cd read_message/
cdk init --language csharp
tree -L 3
.
├── cdk.json
├── README.md
└── src
├── ReadMessage
│ ├── GlobalSuppressions.cs
│ ├── Program.cs
│ ├── ReadMessage.csproj
│ └── ReadMessageStack.cs
└── ReadMessage.sln
2 directories, 7 files
Now, add some templates for the project using dotnet cli.
cd src
dotnet new lambda.SQS -o app/SaveMessage
dotnet sln add .\app\ SaveMessage \src\ SaveMessage
dotnet add .\app\ SaveMessage \src\ SaveMessage package AWSSDK.DynamoDBv2
Finally, For environment properties for this setup are loaded from environment variables. This block is in the main file program.cs
...
Env = new Amazon.CDK.Environment
{
Account = System.Environment.GetEnvironmentVariable("CDK_DEFAULT_ACCOUNT"),
Region = System.Environment.GetEnvironmentVariable("CDK_DEFAULT_REGION"),
}
...
⚠️ The
ReadMessageStack.cs
contains the stack for the microservice architecture, in this case some values are hardcoded, but this isn’t recommended for production environments, ever use properties in a yaml file or json.
Finaly, create the pipeline file:
# File: azure-pipelines.yml
resources:
repositories:
- repository: cdk_pipelines
type: git
name: DevSecOps/cdk_pipelines
trigger:
- master
variables:
- group: cdk_pipelines_serverless_deployments
pool:
vmImage: ubuntu-latest
stages:
- template: templates/ci_cd.yaml@cdk_pipelines # Template reference
parameters:
Listlambdas:
- SaveMessage
AppPath: app
In the next section you'll find the pipeline configuration in deep.
The Figure 2 depicts the final structure for your CDK project.
Azure DevOps Pipeline
Create azure DevOps template’s structure.
For other hand, as a DevSecOps engineer you must grant the compliance, automate deployments, and best practices in the applications code, so the first step is creating a central templates repository for reuse and scale easily, the structure is depicted in the Figure 3, usually these templates are storage and governed in an independent azure project, for this blog its name is DevSecOps:
Figure 3. Azure DevOps pipelines Templates
The template repository is composed of a template folder will have three files:
ci_cd.yaml: define a multistage pipeline with the main steps for synth and deploy application trough different environments up to production.
pull_request.yaml: define simple steps for validate a pull request, apply sast, sca, unit test, and deploy steps into development environment.
features.yaml: define the pipeline for deployment feature scenarios in development environment or account and include a destroy step to reclaim the resources for development purposes clear. Of course, each feature branch must be start with feature/.
Also, in this repository, you can find a common folder that contains the common steps and settings for all CDK projects.
For example the Figure 4 depicts the structure for this repository:
Figure 4. CDK Pipelines Template Repository
For this blog the sast practice for application business code wouldn't applied, in future post could be explained.
For example, the ci_cd.yaml
file content:
# CDK Pipeline
parameters:
- name: Listlambdas
type: object
default:
- ReadMessage
- SaveReadMessage
- name: ServiceConnection
type: string
default: DevSecOpsServelessDeployment
- name: RegionName
type: string
default: us-east-2
- name: AppPath
type: string
default: lambdas
- name: Project
type: string
default: DevSecOpsServelessDeployment
- name: Environment
type: string
default: dev
stages:
- stage: CI
displayName: CIStage
jobs:
- job: SASTIaC
steps:
- template: ../common/install.yaml
- template: ../common/build.yaml
parameters:
Environment: ${{ parameters.Environment }}
RegionName: ${{ parameters.RegionName }}
ServiceConnection: ${{ parameters.ServiceConnection }}
Project: ${{ parameters.Project }}
AppPath: ${{ parameters.AppPath }}
ListLambdas: ${{ parameters.Listlambdas }}
- stage: Deploy
displayName: DeployStage${{ parameters.Environment }}
jobs:
- job: DeployApp
steps:
- checkout: none #skip checking out the default repository resource
- template: ../common/install.yaml
- template: ../common/deploy.yaml
parameters:
Environment: ${{ parameters.Environment }}
RegionName: ${{ parameters.RegionName }}
ServiceConnection: ${{ parameters.ServiceConnection }}
Project: ${{ parameters.Project }}
The common folder contains the main steps for CDK deployment:
- install.yaml: define install steps for CDK cli and tools such checkov. For example:
#CDK Pipelines Common steps
steps:
#Install node js
- task: NodeTool@0
inputs:
versionSpec: '18.x'
displayName: 'Install Node.js'
#Install cdk and chechov for sast
- script: |
npm install -g aws-cdk
pip install checkov
displayName: 'Install'
- build.yaml: define steps for continuous building for CDK project. For example:
#Build steps
# SAST for IaC
parameters:
- name: ServiceConnection
type: string
default: DevSecOpsServelessDeployment
- name: RegionName
type: string
default: us-east-2
- name: Project
type: string
default: DevSecOpsServelessDeployment
- name: Environment
type: string
default: dev
- name: Language
type: string
default: c#
- name: AppPath
type: string
default: lambdas
- name: Listlambdas
type: object
default:
- ReadMessage
- SaveReadMessage
steps:
# dotnet
- ${{ if eq(parameters.Language, 'c#') }}:
- script: |
echo 'Build for environment Account - ' $(${{ parameters.Environment }}_account) 'Region-' $(${{ parameters.Environment }}_region)
dotnet build src
displayName: 'BuildDotnetProject'
env:
CDK_DEPLOY_ACCOUNT: $(${{ parameters.Environment }}_account)
CDK_DEPLOY_REGION: $(${{ parameters.Environment }}_region)
#Compile app for serveless projects
- ${{ each value in parameters.Listlambdas }}:
- script: |
echo ${{ value }}
dotnet build src/${{ parameters.AppPath }}/${{ value }}/src/${{ value }}/
displayName: 'Build Lambdas'
- script: |
cdk synth
checkov -d cdk.out/ -o junitxml --output-file-path . -s #-s flag use soft mode
displayName: 'SASTIaC'
- task: PublishTestResults@2
inputs:
testResultsFormat: 'JUnit' # 'JUnit' | 'NUnit' | 'VSTest' | 'XUnit' | 'CTest'. Alias: testRunner. Required. Test result format. Default: JUnit.
testResultsFiles: '**/results_junitxml.xml'
mergeTestResults: true
failTaskOnFailedTests: false
testRunTitle: 'SAST IaC Checkov scan'
condition: 'always()'
- task: AWSShellScript@1
inputs:
awsCredentials: ${{ parameters.ServiceConnection }}
regionName: ${{ parameters.RegionName }}
scriptType: 'inline'
inlineScript: |
echo 'Diff for environment Account - ' $(${{ parameters.Environment }}_account) 'Region-' $(${{ parameters.Environment }}_region)
cdk diff
displayName: 'CDK Diff'
env:
CDK_DEPLOY_ACCOUNT: $(${{ parameters.Environment }}_account)
CDK_DEPLOY_REGION: $(${{ parameters.Environment }}_region)
# Publish Package
- task: PublishPipelineArtifact@1
displayName: PublishArtifact
inputs:
targetPath: '$(Build.SourcesDirectory)'
artifactName: ${{ parameters.Project }}
- deploy.yaml: define steps for deploying CDK project. For example:
#Deploy app
parameters:
- name: ServiceConnection
type: string
default: DevSecOpsServelessDeployment
- name: RegionName
type: string
default: us-east-2
- name: Project
type: string
default: DevSecOpsServelessDeployment
- name: Environment
type: string
default: dev
steps:
- task: DownloadPipelineArtifact@2
inputs:
artifact: ${{ parameters.Project }}
patterns: '**/*'
path: $(Build.SourcesDirectory)
- script: |
ls -all
pwd
- task: AWSShellScript@1
inputs:
awsCredentials: ${{ parameters.ServiceConnection }}
regionName: ${{ parameters.RegionName }}
scriptType: 'inline'
inlineScript: |
echo 'Deploying in ' $CDK_DEFAULT_ACCOUNT $CDK_DEFAULT_REGION
cdk deploy --require-approval never
displayName: 'CDK Deployement'
env:
CDK_DEPLOY_ACCOUNT: $(${{ parameters.Environment }}_account)
CDK_DEPLOY_REGION: $(${{ parameters.Environment }}_region)
Also, a variable group is created with the account and region values for different environments.
az pipelines variable-group create --name cdk_pipelines_serverless_deployments \
--variables dev_account=123456789014 dev_region=us-east-2 prod_account=123456789015 prod_region=us-east-2 \
--authorize true \
--description "Group variables for serveless deployment demo" \
--org https://dev.azure.com/MyOrg\
--project ServelessDeployment
Now, you must create both repositories and try the deployment. The following commands could help you:
# Create repositories
az repos create --name 'chatapp_read_message' --project ServelessDeployment
# Add remote Origin
git remote add origin https://MyOrg@dev.azure.com/MyOrg/ServelessDeployment/_git/chatapp_read_message
az repos create --name 'cdk_pipelines' --project DevSecOps
#Create pipelines
az pipelines create --name 'chatapp_read_message' --description 'Pipeline for cdk deployment project chatapp_read_message' \
--repository chatapp_read_message --branch master --yml-path azure-pipelines.yml --project ServelessDeployment --repository-type tfsgit
#Run Pipeline
az pipelines run --name chatapp_read_message --project ServelessDeployment
Finally you should get the following results:
Figure 5. Deployment Azure DevOps.
Figure 6. Deployment AWS CloudFormation console.
You can find the yaml templates in:
velez94 / cdkv2_pipelines_azure_devops
Repository with CDK pipelines definitios in yaml files for azure devops.
CDK Pipelines in azure DevOps
This repository contains de CDK pipelines definitions for integrating azure DevOps and AWS, based on simple authentication and multi account setup in AWS.
Architecture Diagram
Requirements
- Bootstrap accounts for CDK deployments.
- A Service Connection with right permissions.
How to
- Create a variable group with values for environments, for example:
az pipelines variable-group create --name cdk_pipelines_delivery --variables dev_account=123456789012 dev_region=us-east-2 --authorize true --description "Group for lab Pipelines Delivery" --project Delivery
- Create an azure pipeline file into cdk project to use these templates, for example:
# File: azure-pipelines-c#.yml
variables:
- group: 'ephemeral_envrionments_demo'
resources:
repositories:
- repository: cdk_pipelines
type: git
name: DevSecOps/cdk_pipelines
trigger:
- master
pool:
vmImage: ubuntu-latest
stages:
- template: templates/ci_cd.yaml@cdk_pipelines # Template reference
parameters:
Project: ephemeral_env_core
Language: typescript
Action: deploy
ServiceConnection: EphemeralEnvDeployment
Learn how in:
Thanks for reading and sharing! 😃
Top comments (0)