DEV Community

Salah Elhossiny for AWS Community Builders

Posted on • Updated on

Deploying CI/CD For NodeJS Serverless Applications Workshop: Part IV

BUILD THE PIPELINE

In this part you are going to learn how to automate the build, package and deploy commands by creating a continous delivery pipeline using AWS Code Pipeline.

image

The services used are in this part:

  • CodeCommit
  • CodeBuild
  • CodePipeline
  • CloudFormation
  • AWS CDK

CREATE A GIT REPOSITORY

Any CI/CD pipeline starts with a code repository.

In this workshop we use AWS CodeCommit for ease of integration, but you could use other source code integrations, like GitHub for example.

Run the following command from your terminal to create a new CodeCommit repository:

aws codecommit create-repository --repository-name sam-app
Enter fullscreen mode Exit fullscreen mode

You should see the following output. Copy the value of cloneUrlHttp, you will need it later.

image

CONFIGURE CREDENTIALS

One of the cool things about CodeCommit is the support for IAM authentication. And if you are running this workshop from a Cloud9 workspace, you can leverage the fact that your terminal is already pre-authenticated with valid AWS credentials.

Configure the git client with username and email, so your commits have an author defined.

~ git config --global user.name "Replace with your name"
~ git config --global user.email "replace_with_your_email@example.com"
Enter fullscreen mode Exit fullscreen mode

PUSH THE CODE

Ignore the build artifacts

Copy/paste the following lines at the end of the sam-app/.gitgnore file.

There is no need to track the .aws-sam directory or the packaged.yaml under version control as they are re-generated on every build.

.aws-sam/
packaged.yaml
Enter fullscreen mode Exit fullscreen mode

From the root directory of your sam-app project, run the following commands:

~ cd ~/environment/sam-app
~ git init
~ git add .
~ git commit -m "Initial commit"
Enter fullscreen mode Exit fullscreen mode

Push the code

Add your CodeCommit repository URL as a remote on your local git project.

~ git remote add origin codecommit://sam-app
Enter fullscreen mode Exit fullscreen mode

Now, push the code:

~ git push -u origin master
Enter fullscreen mode Exit fullscreen mode

Verify in CodeCommit

Navigate to the AWS CodeCommit console, find your sam-app repository and click on it to view its contents.

Make sure your code is there. You should see a screen like the following:

image

HOW TO BUILD A PIPELINE

The best way to automate the creation of CI/CD pipelines is by provisioning them programmatically via Infrastructure as Code.

This is specially useful in a microservices environment, where you have a pipeline per microservice, which potentially means dozens of pipelines, if not more.

Having an automated way to create these pipelines enables developers to create as many as necessary without building them manually from the console every time.

Different ways to create pipelines

Different mechanisms are used for creating pipelines programmatically. Nowadays developers have many choices to pick from, but the most common ones we see are the following:

  • AWS CloudFormation
  • AWS CDK
  • Terraform
  • AWS Serverless App Repository

Introducing the AWS CDK

In this workshop, we are going to use the AWS Cloud Development Kit (also known as CDK), as the pipeline vending mechanism.

The AWS CDK is a software development framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation.

That’s right! You can describe your infrastructure by writing code in TypeScript, C#, Python or Java.

Your code is then synthesized into CloudFormation and using the CDK CLI you can deploy it to an AWS account.

How do SAM and CDK play together?

Serverless developers use the SAM framework to define their applications, SAM CLI to build them and deploy them and AWS CDK to provision any infrastructure related resources, like their CI/CD Pipeline.

The nice thing about these tools is that they all share a common ground: CloudFormation.

SETUP A CDK PROJECT

Now, let’s create a folder within our sam-app directory where the pipeline code will reside.

 ~ cd ~/environment/sam-app
 ~ mkdir pipeline
 ~ cd pipeline
Enter fullscreen mode Exit fullscreen mode

Initialize a new CDK project within the pipeline folder by running the following command:

cdk init --language typescript
Enter fullscreen mode Exit fullscreen mode

Now install the CDK modules that we will be using to build a pipeline:

npm install --save @aws-cdk/aws-codedeploy @aws-cdk/aws-codebuild
npm install --save @aws-cdk/aws-codecommit @aws-cdk/aws-codepipeline-actions
npm install --save @aws-cdk/aws-s3
Enter fullscreen mode Exit fullscreen mode

Project structure

At this point, your project should have the structure below (only the most relevant files and folders are shown).

Within the CDK project, the main file you will be interacting with is the pipeline-stack.ts. Don’t worry about the rest of the files for now.

sam-app                         # SAM application root
├── hello-world                 # Lambda code
├── samconfig.toml              # Config file for manual deployments
├── template.yaml               # SAM template
└── pipeline                    # CDK project root
    └── lib
        └── pipeline-stack.ts   # Pipeline definition
    └── bin
        └── pipeline.ts         # Entry point for CDK project
    ├── cdk.json
    ├── tsconfig.json
    ├── package.json
    └── jest.config.js
Enter fullscreen mode Exit fullscreen mode

PIPELINE AS CODE

Open the file lib/pipeline-stack.ts in your Cloud9 workspace. It is empty at the moment, but here is where you will be adding code to build your CI/CD pipeline.

image

Build the CDK project

Even though we haven’t wrote any code yet, let’s get familiar with how to build and deploy a CDK project, as you will be doing it multiple times in this workshop and you should get comfortable with the process. Start by building the project with the following command:

cd ~/environment/sam-app/pipeline
npm run build
Enter fullscreen mode Exit fullscreen mode

Deploy a CDK project

After the build has finished, go ahead and deploy the pipeline project by using the CDK CLI:

cdk deploy
Enter fullscreen mode Exit fullscreen mode

The output should look like the following:

image

A new CloudFormation stack was created in your account, but because your CDK project is empty, the only resource that was created was an AWS::CDK::Metadata.

ARTIFACTS BUCKET

Every Code Pipeline needs an artifacts bucket, also known as Artifact Store. CodePipeline will use this bucket to pass artifacts to the downstream jobs and its also where SAM will upload the artifacts during the build process.

Let’s get started and write the code for creating this bucket:

Make sure you are editing the pipeline-stack file with .ts extension

// lib/pipeline-stack.ts

import * as cdk from '@aws-cdk/core';
import s3 = require('@aws-cdk/aws-s3');
import codecommit = require('@aws-cdk/aws-codecommit');
import codepipeline = require('@aws-cdk/aws-codepipeline');
import codepipeline_actions = require('@aws-cdk/aws-codepipeline-actions');
import codebuild = require('@aws-cdk/aws-codebuild');

export class PipelineStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // The code that defines your stack goes here
    const artifactsBucket = new s3.Bucket(this, "ArtifactsBucket");
  }
}

Enter fullscreen mode Exit fullscreen mode

Easy right? Now build and deploy the project like you did it earlier:

npm run build
cdk deploy
Enter fullscreen mode Exit fullscreen mode

Note

If you get a build error, check that all the @aws-cdk dependencies in the package.json file have the same version number, if not, fix it, delete the node_modules folder and run npm install.

The output will show that the S3 bucket got created:

image

SOURCE STAGE

The Source Stage is the first step of any CI/CD pipeline and it represents your source code. This stage is in charge of triggering the pipeline based on new code changes (i.e. git push or pull requests). In this workshop, we will be using AWS CodeCommit as the source provider, but CodePipeline also supports S3, GitHub and Amazon ECR as source providers.

Append the following code snippet after your bucket definition in the pipeline-stack.ts file:

// Import existing CodeCommit sam-app repository
const codeRepo = codecommit.Repository.fromRepositoryName(
  this,
  'AppRepository', // Logical name within CloudFormation
  'sam-app' // Repository name
);

// Pipeline creation starts
const pipeline = new codepipeline.Pipeline(this, 'Pipeline', {
  artifactBucket: artifactsBucket
});

// Declare source code as an artifact
const sourceOutput = new codepipeline.Artifact();

// Add source stage to pipeline
pipeline.addStage({
  stageName: 'Source',
  actions: [
    new codepipeline_actions.CodeCommitSourceAction({
      actionName: 'CodeCommit_Source',
      repository: codeRepo,
      output: sourceOutput,
    }),
  ],
});

Enter fullscreen mode Exit fullscreen mode

Notice how we define an object sourceOutput as a pipeline artifact; this is necessary for any files that you want CodePipeline to pass to downstream stages.

In this case, we want our source code to be passed to the Build stage.

Info

Don’t do a cdk deploy just yet, because a pipeline needs to have at least 2 stages to be created. Lets continue to the next page to add a Build stage first.

BUILD STAGE

The Build Stage is where your Serverless application gets built and packaged by SAM. We are going to use AWS CodeBuild as the Build provider for our pipeline.

It is worth mentioning that CodePipeline also supports other providers like Jenkins, TeamCity or CloudBees.

AWS CodeBuild is a great option because you only pay for the time where your build is running, which makes it very cost effective compared to running a dedicated build server 24 hours a day when you really only build during office hours.

It is also container-based which means that you can bring your own Docker container image where your build runs, or use a managed image provided by CodeBuild.

Add the build stage

Let’s go ahead and add a Build stage to you pipeline-stack.ts:


// Declare build output as artifacts
const buildOutput = new codepipeline.Artifact();

// Declare a new CodeBuild project
const buildProject = new codebuild.PipelineProject(this, 'Build', {
  environment: { buildImage: codebuild.LinuxBuildImage.AMAZON_LINUX_2_2 },
  environmentVariables: {
    'PACKAGE_BUCKET': {
      value: artifactsBucket.bucketName
    }
  }
});

// Add the build stage to our pipeline
pipeline.addStage({
  stageName: 'Build',
  actions: [
    new codepipeline_actions.CodeBuildAction({
      actionName: 'Build',
      project: buildProject,
      input: sourceOutput,
      outputs: [buildOutput],
    }),
  ],
});

Enter fullscreen mode Exit fullscreen mode

Deploy the pipeline

From your terminal, run the following commands to deploy the pipeline:

npm run build
cdk deploy
Enter fullscreen mode Exit fullscreen mode

Verify pipeline creation

Navigate to the AWS CodePipeline Console and click on your newly created pipeline!

image

The Build step should have failed. Don’t worry! this is expected because we haven’t specified what commands to run during the build yet, so AWS CodeBuild doesn’t know how to build our Serverless application.

image

Now, let's fix it!

BUILDSPEC FILE

A Buildspec File is a series of commands in YAML format that CodeBuild executes to build your application.

This file is placed in the root folder of a SAM application and CodeBuild will automatically find it and run it during build time.

In the root folder of app, create file named "buildspec.yml" and paste the following in it:

# ~/environment/sam-app/buildspec.yml

version: 0.2
phases:
  install:
    runtime-versions:
      nodejs: 12
    commands:
      # Install packages or any pre-reqs in this phase.
      # Upgrading SAM CLI to latest version
      - pip3 install --upgrade aws-sam-cli
      - sam --version
      # Installing project dependencies
      - cd hello-world
      - npm install

  pre_build:
    commands:
      # Run tests, lint scripts or any other pre-build checks.
      - npm run test

  build:
    commands:
      # Use Build phase to build your artifacts (compile, etc.)
      - cd ..
      - sam build

  post_build:
    commands:
      # Use Post-Build for notifications, git tags, upload artifacts to S3
      - sam package --s3-bucket $PACKAGE_BUCKET --output-template-file packaged.yaml

artifacts:
  discard-paths: yes
  files:
    # List of local artifacts that will be passed down the pipeline
    - packaged.yaml

Enter fullscreen mode Exit fullscreen mode

Take a moment to understand the structure of the file and feel free to read the Buildsec Reference here:

https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html

Commit your changes and push them to the repository.

cd ~/environment/sam-app
git add .
git commit -m "Added buildspec.yml"
git push
Enter fullscreen mode Exit fullscreen mode

Navigate to your CodePipeline again, and wait for it to trigger automatically. This time the build will succeed:

image

DEPLOY STAGE

The Deploy Stage is where your SAM application and all its resources are created an in an AWS account.

The most common way to do this is by using CloudFormation ChangeSets to deploy. This means that this stage will have 2 actions: the CreateChangeSet and the ExecuteChangeSet.

Add the Deploy stage to your pipeline-stack.ts:

// Deploy stage
pipeline.addStage({
  stageName: 'Dev',
  actions: [
    new codepipeline_actions.CloudFormationCreateReplaceChangeSetAction({
      actionName: 'CreateChangeSet',
      templatePath: buildOutput.atPath("packaged.yaml"),
      stackName: 'sam-app',
      adminPermissions: true,
      changeSetName: 'sam-app-dev-changeset',
      runOrder: 1
    }),
    new codepipeline_actions.CloudFormationExecuteChangeSetAction({
      actionName: 'Deploy',
      stackName: 'sam-app',
      changeSetName: 'sam-app-dev-changeset',
      runOrder: 2
    }),
  ],
});

Enter fullscreen mode Exit fullscreen mode

Deploy the pipeline

On your terminal, run the following commands from within the pipeline directory:

cd ~/environment/sam-app/pipeline
npm run build
cdk deploy
Enter fullscreen mode Exit fullscreen mode

Navigate to your pipeline and you will see the Deploy stage has been added, however, it is currently grayed out because it hasn’t been triggered. Let’s just trigger a new run of the pipeline manually by clicking the Release Change buttton.

image

In the terminal, run the following commands from the root directory of your sam-app project.

git add .
git commit -m "CI/CD Pipeline definition"
git push
Enter fullscreen mode Exit fullscreen mode

Congratulations! You have created a CI/CD pipeline for a Serverless application!

Discussion (0)