DEV Community

Deploying CI/CD For NodeJS Serverless Applications Workshop: Part V

CANARY DEPLOYMENTS

A Canary Deployment is a technique that reduces the risk of deploying a new version of an application by slowly rolling out the changes to a small subset of users before rolling it out to the entire customer base.

image

HOW DOES IT WORK?

The concepts of blue/green and canary deployments have been around for a while and have been well-established as best-practices for reducing the risk of software deployments. In traditional applications, you slowly and incrementally update the servers in your fleet while simultaneously verifying application health. However, there is somewhat of an impedance mismatch when mapping these concepts to a serverless world. You can’t incrementally deploy your software across a fleet of servers when there are no servers!

The answer is that there are a couple of services and features involved in making this possible. Let us explain:

Lambda versions and aliases

AWS Lambda allows you to publish multiple versions of the same function. Each version has its own code and associated dependencies, as well as its own function settings (like memory allocation, timeout and environment variables).

You can then refer to a given version by using a Lambda Alias. An alias is nothing but a name that can be pointed to a given version of a Lambda function.

image

Traffic shifting with Lambda aliases

With the introduction of alias traffic shifting, it is now possible to trivially implement canary deployments of Lambda functions. By updating additional version weights on an alias, invocation traffic is routed to the new function versions based on the weight specified. Detailed CloudWatch metrics for the alias and version can be analyzed during the deployment, or other health checks performed, to ensure that the new version is healthy before proceeding.

image

Traffic shifting with SAM and CodeDeploy

AWS CodeDeploy provides an intuitive turn-key implementation of this functionality integrated directly into AWS SAM.

Traffic-shifted deployments can be declared in a SAM template, and CodeDeploy manages the function rollout as part of the CloudFormation stack update. CloudWatch alarms can also be configured to trigger a stack rollback if something goes wrong.

image

UPDATE SAM TEMPLATE

Open the SAM template (sam-app/template.yaml) in your project and add the following lines to the HelloWorldFunction properties section.

AutoPublishAlias: live
DeploymentPreference:
    Type: Canary10Percent5Minutes

Enter fullscreen mode Exit fullscreen mode

Take care of indentation, YAML files ! :D

image

Deployment Preference Types

For this workshop, we are using the Canary10Percent5Minutes strategy, which means that traffic is shifted in two increments. In the first increment, only 10% of the traffic is shifted to the new Lambda version, and after 5 minutes, the remaining 90% is shifted.

There are other deployment strategies you can choose in CodeDeploy:

  • Canary10Percent30Minutes
  • Canary10Percent5Minutes
  • Canary10Percent10Minutes
  • Canary10Percent15Minutes
  • Linear10PercentEvery10Minutes
  • Linear10PercentEvery1Minute
  • Linear10PercentEvery2Minutes
  • Linear10PercentEvery3Minutes
  • AllAtOnce

The Linear strategy means that traffic is shifted in equal increments with an equal number of time interval between each increment.

Validate the SAM template

Run the following command on your terminal:

~ cd ~/environment/sam-app
~ sam validate
Enter fullscreen mode Exit fullscreen mode

If the template is correct, you will see template.yaml is a valid SAM Template. If you see an error, then you likely have an indentation issue on the YAML file. Double check and make sure it matches the screenshot shown above.

Push the changes

In the terminal, run the following commands from the root directory of your sam-app project.

~ git add .
~ git commit -m "Canary deployments with SAM"
~ git push
Enter fullscreen mode Exit fullscreen mode

MONITOR CANARY HEALTH

Canary deployments are considerably more successful if the code is being monitored during the deployment. You can configure CodeDeploy to automatically roll back the deployment if a specified CloudWatch metric has breached the alarm threshold. Common metrics to monitor are Lambda Invocation errors or Invocation Duration (latency), for example.

Define a CloudWatch Alarm

Add the following alarm definition to the template.yaml file in the Resources section after the HelloWorldFunction definition.

CanaryErrorsAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmDescription: Lambda function canary errors
    ComparisonOperator: GreaterThanThreshold
    EvaluationPeriods: 2
    MetricName: Errors
    Namespace: AWS/Lambda
    Period: 60
    Statistic: Sum
    Threshold: 0
    Dimensions:
      - Name: Resource
        Value: !Sub "${HelloWorldFunction}:live"
      - Name: FunctionName
        Value: !Ref HelloWorldFunction
      - Name: ExecutedVersion
        Value: !GetAtt HelloWorldFunction.Version.Version

Enter fullscreen mode Exit fullscreen mode

And then add the following lines to the DeploymentPreference section of the HelloWorldFunction definition.

Alarms:
    - !Ref CanaryErrorsAlarm
Enter fullscreen mode Exit fullscreen mode

Your template.yaml should look like this:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app

  Sample SAM Template for sam-app

Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs14.x
      AutoPublishAlias: live
      DeploymentPreference:
        Type: Canary10Percent5Minutes
        Alarms:
          - !Ref CanaryErrorsAlarm
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

  CanaryErrorsAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: Lambda function canary errors
      ComparisonOperator: GreaterThanThreshold
      EvaluationPeriods: 2
      MetricName: Errors
      Namespace: AWS/Lambda
      Period: 60
      Statistic: Sum
      Threshold: 0
      Dimensions:
        - Name: Resource
          Value: !Sub "${HelloWorldFunction}:live"
        - Name: FunctionName
          Value: !Ref HelloWorldFunction
        - Name: ExecutedVersion
          Value: !GetAtt HelloWorldFunction.Version.Version

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn
Enter fullscreen mode Exit fullscreen mode

Then you can validate again:

cd ~/environment/sam-app
sam validate
Enter fullscreen mode Exit fullscreen mode

If it's ok, then push again:

git add .
git commit -m "Added CloudWatch alarm to monitor the canary"
git push
Enter fullscreen mode Exit fullscreen mode

Navigate to the AWS CodeDeploy console and after a couple of minutes, you should see a new deployment in progress. Click on the Deployment to see the details.

image

The deployment status shows that 10% of the traffic has been shifted to the new version (aka The Canary). CodeDeploy will hold the remaining percentage until the specified time interval has ellapsed, in this case we specified the interval to be 5 minutes.

image

Shortly after the 5 minutes, the remaining traffic should be shifted to the new version:

image

ROLLBACKS

Monitoring the health of your canary allows CodeDeploy to make a decision to whether a rollback is needed or not. If any of the CloudWatch Alarms specified gets to ALARM status, CodeDeploy rollsback the deployment automatically.

Introduce an error on purpose

Lets break the Lambda function on purpose so that the CanaryErrorsAlarm gets triggered during deployment. Update the lambda code in sam-app/hello-world/app.js to throw an error on every invocation, like this:


let response;

exports.lambdaHandler = async (event, context) => {
    throw new Error("This will cause a deployment rollback");
    // try {
    //     response = {
    //         'statusCode': 200,
    //         'body': JSON.stringify({
    //             message: 'hello my friend with canaries',
    //         })
    //     }
    // } catch (err) {
    //     console.log(err);
    //     return err;
    // }

    // return response
};
Enter fullscreen mode Exit fullscreen mode

Make sure to update the unit test, otherwise the build will fail. Comment out every line in the sam-app/hello-world/tests/unit/test-handler.js file:

// 'use strict';

// const app = require('../../app.js');
// const chai = require('chai');
// const expect = chai.expect;
// var event, context;

// describe('Tests index', function () {
//     it('verifies successful response', async () => {
//         const result = await app.lambdaHandler(event, context)

//         expect(result).to.be.an('object');
//         expect(result.statusCode).to.equal(200);
//         expect(result.body).to.be.an('string');

//         let response = JSON.parse(result.body);

//         expect(response).to.be.an('object');
//         expect(response.message).to.be.equal("hello my friend with canaries");
//     });
// });
Enter fullscreen mode Exit fullscreen mode

In the terminal, run the following commands from the root directory of your sam-app project.

git add .
git commit -m "Breaking the lambda function on purpose"
git push

Enter fullscreen mode Exit fullscreen mode

Again, wait for your Pipeline to reach the deployment phase (ExecuteChangeSet). It should turn blue when it begins:

image

INVOKE THE CANARY

While the deployment is running, you need to generate traffic to the new Lambda function to make it fail and trigger the CloudWatch Alarm. In a real production environment, your users will likely generate organic traffic to the canary function, so you may not need to do this.

In your terminal, run the following command to invoke the Lambda function:

aws lambda invoke --function-name \
$(aws lambda list-functions | jq -r -c '.Functions[] | select( .FunctionName | contains("sam-app-HelloWorldFunction")).FunctionName'):live \
--payload '{}'  response.json

Enter fullscreen mode Exit fullscreen mode

image

Remember: During deployment, only 10% of the traffic will be routed to the new version. So, keep on invoking your lambda many times. 1 out of 10 invocations should trigger the new broken lambda, which is what you want to cause a rollback.

Here is a command that invokes your function 15 times in a loop. Feel free to run it in your terminal.

counter=1
while [ $counter -le 15 ]
do
    aws lambda invoke --function-name \
    $(aws lambda list-functions | jq -r -c '.Functions[] | select( .FunctionName | contains("sam-app-HelloWorldFunction")).FunctionName'):live \
    --payload '{}' \
    response.json
    sleep 1
    ((counter++))
done

Enter fullscreen mode Exit fullscreen mode

Navigate to the AWS CodeDeploy Console and go into the deployment In-Progress to view its details.

After a few minutes, CodeDeploy will detect that the CanaryErrorsAlarm has triggered and it will start rolling back the deployment. The screen will look something like this:

image

CLEANUP

DELETE S3 BUCKETS

Empty the buckets by going to the S3 console. And then Delete them (there are 2 buckets).

image

image

DELETE CF STACKS

Now that the buckets are empty, we can delete the Cloudformation stacks:

Delete the CF stacks in following sequence and please wait for each to complete before going to next one.

~ aws cloudformation delete-stack --stack-name sam-app

~ aws cloudformation delete-stack --stack-name sam-app-cicd

~ aws cloudformation delete-stack --stack-name aws-sam-cli-managed-default

Enter fullscreen mode Exit fullscreen mode

We can now delete the Cloud9 instance we created:

  • Go to your Cloud9 Environment.
  • Select the environment (i.e. MyCloud9Workspace) and click delete.

image

Congratulations, you finished the workshop. Thank you for reading all parts of this workshop series.

Discussion (0)