Background
When starting new projects with clients, the first thing I often do is to set a foundation, a so-called baseline. We discuss our team/DevOps agreements, some technical ones like; "Thou shalt not push to main" and others more on a process level, like we do scrum etc. When those agreements are signed in blood by everyone, the actual work starts. From a technical point of view, it starts with creating a repository.
As I am an AWS fanboy, and helping clients with their Cloud journey, I try to stick to native AWS services as much as possible. This means for code storing I use AWS CodeCommit as a git repository, not always preferred, but in enterprises often mandatory. As I am also a CDK lover, I combine the AWS services with CDK pipelines, see also the previous post on CDK pipelines.
Back to the DevOps agreements, one of our rules is that you should not push to main and everything which needs to end up in main, needs to be peer reviewed by a colleague. So how can we tackle this via code?
Well first of all, on the role used by developers, we set a refs condition with deny permissions (sometimes you need the platform team for that), so users can't directly push to the main branch anymore:
CodecommitPolicy:
Type: AWS::IAM::ManagedPolicy
Properties:
Description: Codecommit policy
ManagedPolicyName: landingzone-platform-operator-codecommit-policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: ProtectMainBranch
Effect: Deny
Action:
- codecommit:GitPush
- codecommit:DeleteBranch
- codecommit:PutFile
- codecommit:MergeBranchesByFastForward
- codecommit:MergeBranchesBySquash
- codecommit:MergeBranchesByThreeWay
- codecommit:MergePullRequestByFastForward
- codecommit:MergePullRequestBySquash
- codecommit:MergePullRequestByThreeWay
Resource: <YOUR REPO ARN>
Condition:
StringEqualsIfExists:
codecommit:References:
- refs/heads/main
'Null':
codecommit:References: 'false'
With this policy in place, users need to create Pull Requests (PR's). Because we also want to prohibit users from approving their PRs, some extra configuration is needed.
CodeCommit can use approval templates for that. What is even nicer is that you can also trigger a CodeBuild project to for example run tests or validations on your code. How to do that in CDK I will describe below.
Real World Scenario
So within the DevOps Agreements, we have stated 3 important rules:
- You shall not push to the main branch directly, but create Pull Requests.
- Your Pull Request needs to be verified by a colleague, also called a peer review.
- Your code is tested
As we already mentioned how to limit users to directly push to the main branch, I will skip rule number 1. Focussing on numbers 2 and 3, let's create a CDK app which supports peer reviewing and testing. What we want to achieve is explained in the following diagram:
A Developer pushes to a feature branch, and depending on which branch the pull request is created, develop or main branch, several approvals need to be given. The creation of a pull request triggers an event which kicks off a CodeBuild project which will run tests and linting. When everything passes, approval will be given by the CodeChecker and you only need one approval from your colleague to continue the merge to main.
GO Build
Start with creating a CDK app. My example is in Python:
❯ cdk init app --language=python
On constructs.dev there is a cool construct available which takes away a lot of burden to create all the templates. So let's use that package. Install and import it in the cdk app.
❯ pip3 install cloudcomponents.cdk_pull_request_check
❯ pip install cloudcomponents.cdk_pull_request_approval_rule
When the package is installed, don't forget to put it in your requirements.txt. Oke let's break down the code. Below you see the start of the codechecker_stack.py file. Here we do all the imports, including the installed cloudcomponents construct from constructs.dev.
from aws_cdk import (
Aspects,
Stack,
aws_iam as iam,
aws_codecommit as codecommit,
aws_codebuild as codebuild,
aws_kms,
aws_events_targets as event_target,
aws_sns as sns,
custom_resources as custom_resources,
)
from constructs import Construct
from cloudcomponents.cdk_pull_request_check import PullRequestCheck
from cloudcomponents.cdk_pull_request_approval_rule import (
Approvers,
ApprovalRuleTemplate,
ApprovalRuleTemplateRepositoryAssociation,
Template,
)
import cdk_nag
# Constant variables
REPOSITORY_NAME = "my_demo_repo"
MY_ASSUME_ROLE = "MY_ASSUME_ROLE"
APPROVALS_REQUIRED_MAIN = 1
TEMPLATES_FOLDER = "cdk.out"
TEMPLATES_FILE_SUFFIX = "*.template.json"
There are some constant variables to fill in. Let's go by them one by one:
- REPOSITORY_NAME: the repository where you want to enable the codechecker on.
- MY_ASSUME_ROLE: the role which you want to allow to also approve your PR.
- APPROVALS_REQUIRED_MAIN: for how many approvals need to be given to allow a merge.
- TEMPLATES_FOLDER: This is where your
CloudFormationtemplates are located. - TEMPLATES_FILE_SUFFIX: How does your file naming ends?
class CodecheckerStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
# Import the CodeCommit repository where the codechecker needs to check the pull requests.
self.repository = codecommit.Repository.from_repository_name(
self, "Repository", repository_name=REPOSITORY_NAME
)
# Create the pull request check
pull_request_check = self.create_pull_request_check(self.repository)
# Create SNS topic for monitoring purpose
monitoring_topic = sns.Topic(
self,
"MonitoringTopic",
master_key=aws_kms.Key(
self,
"MonitoringTopicKey",
enable_key_rotation=True,
alias="/sns/monitoring",
),
)
# Sent notification on failed build
pull_request_check.on_check_failed(
"Failed", target=event_target.SnsTopic(monitoring_topic)
)
# Dict for branches and amount of approvals needed via constant variables
approvals_per_branch = {
"main": APPROVALS_REQUIRED_MAIN,
}
# Starting for loop over dict and create approval templates
for branch, required_approvals in approvals_per_branch.items():
create_approval_template = self.create_approval_template(
id=f"{branch}-{required_approvals}",
branch=branch,
required_approvals=required_approvals,
repository_name=self.repository.repository_name,
pr_check_approval_role=pull_request_check.code_build_result_function.role.role_name,
)
# Associate approval rule template with repositories from list
approval_template_association = (
self.associate_approval_rule_template_with_repositories(
id=f"associate-existing-repositories-{branch}-{required_approvals}",
branch=branch,
required_approvals=required_approvals,
)
)
approval_template_association.node.add_dependency(create_approval_template)
# Adding CDK Nag aspects
Aspects.of(self).add(cdk_nag.AwsSolutionsChecks())
The next part is the actual class CodecheckerStack, so let's explain what the code does. First, create the repository. Then create the pull request check by calling the function self.create_pull_request_check, which will be explained in a minute. There will also be an SNS topic created, which later can be used to hook for example Slack notifications via AWS Chatbot. The failed build notifications are sent to this SNS topic. With the dict approvals_per_branch, it is possible to specify per branch how many approvals are needed. In the above example, the main branch only requires 1 approval. But it is also possible to extend this with checks for a develop branch.
With the cloud components construct, the Codebuild project will, when successfully run, update the PR with an auto approval by one. For our use case, this means that you are allowed to merge the PR.
With the for loop, we create per item in the dict an approval template by calling the function self.create_approval_template and associate it with the branch with the function self.associate_approval_rule_template_with_repositories. As extra a dependency is added between the association and the approval template creation.
So now let's dive into the functions. First the create_pull_request_check function:
def create_pull_request_check(
self, repository: codecommit.Repository
) -> PullRequestCheck:
"""Function to create the pull request check, CodeBuild project"""
# Create a CodeBuild project which will check the code on a pull request.
pullrequest = PullRequestCheck(
self,
"PullRequestCheck",
repository=repository,
role=self.create_pull_request_role(
repository_name=repository.repository_name
),
build_image=codebuild.LinuxBuildImage.STANDARD_6_0,
environment_variables={
"TEMPLATES_FOLDER": codebuild.BuildEnvironmentVariable(
type=codebuild.BuildEnvironmentVariableType.PLAINTEXT,
value=TEMPLATES_FOLDER,
),
"TEMPLATES_FILE_SUFFIX": codebuild.BuildEnvironmentVariable(
type=codebuild.BuildEnvironmentVariableType.PLAINTEXT,
value=TEMPLATES_FILE_SUFFIX,
),
},
build_spec=codebuild.BuildSpec.from_object(
{
"version": "0.2",
"env": {"git-credential-helper": "yes"},
"phases": {
"install": {
"commands": [
"npm install -g aws-cdk",
"pip install -r requirements.txt",
"pip3 install cfn-lint",
"gem install cfn-nag",
]
},
"build": {
"commands": [
"cdk synth",
"echo ### Scan templates with cfn-lint ###"
"for template in $(find ./$TEMPLATES_FOLDER -type f -maxdepth 3 -name $TEMPLATES_FILE_SUFFIX); do cfn-lint $template -i W2001 -i W3005 -i E3030; done",
"echo ### Scan templates with cfn-nag ###"
"for template in $(find ./$TEMPLATES_FOLDER -type f -maxdepth 3 -name $TEMPLATES_FILE_SUFFIX); do cfn_nag_scan -i $template; done",
]
},
"post_build": {"commands": ["pytest -v"]},
},
}
),
)
return pullrequest
This function, as the name already suggests, creates the actual pull request CodeBuild project. What it does is install the requirement.txt file, cfn-lint and cfn-nag in a standard CodeBuild project. In the build part, the templates are synthezised and pytest is running. Then the output of the templates is checked with cfn-lint and cfn_nag_scan for template warnings and security issues. This is fully adjustable to your needs. If you want to embed for example extra security checks, this is the spot to do so.
What is important here is how your templates are structured. This example uses CDK with cdk.out folder, but in our deployment framework CloudFormation native is used, for that example the CodeChecker TEMPLATES_FOLDER variable would point to the folder templates, including subfolders, to check the code.
def create_approval_template(
self,
id: str,
branch: str,
required_approvals: int,
repository_name: str,
pr_check_approval_role: str,
) -> ApprovalRuleTemplate:
"""Function to create the actual approval template"""
return ApprovalRuleTemplate(
self,
f"{id}-ApprovalRuleTemplate",
approval_rule_template_name=self.get_approval_rule_template_name(
required_approvals, self.repository.repository_name, branch
),
approval_rule_template_description=f"Requires {required_approvals} approvals from the team to approve the pull request",
template=Template(
approvers=Approvers(
number_of_approvals_needed=required_approvals,
approval_pool_members=[
f"arn:aws:sts::{Stack.of(self).account}:assumed-role/<MY_ASSUMED_ROLE>/*",
f"arn:aws:sts::{Stack.of(self).account}:assumed-role/{pr_check_approval_role}/*",
],
),
branches=[branch],
),
)
@staticmethod
def get_approval_rule_template_name(
required_approvals: int, repository_name: str, branch: str
) -> str:
return f"{str(required_approvals)}-approval-for-{repository_name}-{branch}"
The function create_approval_template (see above) is responsible for creating the CodeCommit approval rules for pull requests in AWS. The template itself states which members (approval_pool_members) are allowed to approve and how many approvals need to be given to allow the pull request to be merged. There is an extra function to return the approval rule name which later is used in the association. Lastly, It returns the approval rule template which is used inside the for loop over the dicts.
def associate_approval_rule_template_with_repositories(
self,
id: str,
branch: str,
required_approvals: int,
) -> ApprovalRuleTemplateRepositoryAssociation:
"""Function to associate the approval template with the repository"""
return ApprovalRuleTemplateRepositoryAssociation(
self,
f"{id}-ApprovalRuleTemplateRepositoryAssociation",
approval_rule_template_name=self.get_approval_rule_template_name(
required_approvals, self.repository.repository_name, branch
),
repository=self.repository,
)
The function associate_approval_rule_template_with_repositories above is responsible for associating the approval template with the repository. Additionally, there is a function create_pull_request_role to be used CodeBuild.
def create_pull_request_role(self, repository_name: str) -> iam.IRole:
role = iam.Role(
self,
"CodeBuildRole",
assumed_by=iam.ServicePrincipal("codebuild.amazonaws.com"),
inline_policies={
"CloudWatchLoggingAccess": iam.PolicyDocument(
statements=[
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=[
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
],
resources=[
f"arn:aws:logs:{Stack.of(self).region}:{Stack.of(self).account}:log-group:/aws/codebuild/{repository_name}-pull-request:*",
f"arn:aws:logs:{Stack.of(self).region}:{Stack.of(self).account}:log-group:/aws/codebuild/{repository_name}-pull-request",
],
),
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=[
"codebuild:BatchPutCodeCoverages",
"codebuild:BatchPutTestCases",
"codebuild:CreateReport",
"codebuild:CreateReportGroup",
"codebuild:UpdateReport",
],
resources=[
f"arn:aws:codebuild:{Stack.of(self).region}:{Stack.of(self).account}:report-group/{repository_name}-pull-request-*"
],
),
]
),
},
)
return role
Summary
With this, we have created a pull request mechanism which is automatically started when someone creates a pull request. The pull request fires off a CodeBuild project where tests and security checks are executed. When everything is successful, the pull request will receive approval automatically. Depending on the amount of approvals that need to be given, the pull request can be merged. It is easily adjustable with extra checks, by adding more options inside the codebuild project.
Top comments (0)