Deploying Docker Containers to Lambda using AWS SAM and CodePipeline (Part 1)

#aws #communitybuilder

Deploying to AWS lambda has many advantages, covering cost and speed of execution. There are many options for deployment, including through the console, AWS CLI, CloudFormation and AWS SAM. However, the standard Lambda package size has a limit of 50mb zipped and 250mb unzipped, which in some cases can cause issues, especially with some machine learning frameworks, where the dependencies can hit this limit before you've even written any code.

To solve this issue, you can deploy your code inside a docker container, which has an effective limit of up to 10GB.

Part 1 of this solution will go through a number of AWS Services to show how to use AWS SAM to deploy a docker container to AWS Lambda, based on the Hello World Template.

Part 2 will show how to add this to a CodeBuild and CodePipeline and deploy automatically via GitHub.

Part 1 will use:

AWS SAM
AWS Lambda
AWS API Gateway
Docker

Part 2 will then use:

Github
AWS Github App
AWS CodeStar Connection
AWS Code Pipeline
AWS CodeBuild
AWS CloudFormation

This is what our deployment will look like after both parts are completed:

The first step is to set up AWS Access Keys: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html

Then once you have the Access Key and Secret Key, you need to set them up in a terminal, by either setting up a profile, the AWS CLI directly using 'aws configure' or Environment Variables.

The core of this deployment method is to use the AWS SAM CLI. This deployment can be started with the sample hello world application.

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html

You will need the AWS CLI and the AWS SAM CLI:

AWS CLI

AWS SAM CLI

You can also create a blank application, however some of the template.yaml file from the hello world sample application will be useful, and we'll be using it as the base of this guide.

#Step 1 - Download a sample application
sam init

#Step 2 - Build your application
cd sam-app
sam build

#Step 3 - Deploy your application
sam deploy --guided

This will deploy your Lambda Application and create an API Gateway for you. The guided deploy will have some options to choose, make sure you allow role creation, and at this step, allow deployment without authorisation (which you should set up later if you don't want a public API) as sam will not deploy without it.

You could stop here, but this guide will show you how to turn the Lambda deployment into a Docker Container, and then automate the deployment through the use of Code Pipeline, Code Build and link it to Github.

Within the sam-app folder that is generated, the first file you will need to update is the template.yaml file. The section that creates the Lambda Function is with the type AWS::Serverless::Function.

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.9
      Architectures:
        - x86_64
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

This deploys the Lambda as a python 3.9 application and zipped up, with the x86_64 architecture.

Containerising the Lambda

To change this to use docker we need to make some changes, and define an external docker_file. The main changes are the PackageType, ImageConfig and Metadata. We also remove the CodeUri, Handler and runtime as these are defined in the docker file.

We've also added a HelloWorldAPI section for the API, though one is implicitly defined. If you leave this out you will see some warnings about reserved keywords. This section is also where you can add Cors later on if needed.

Resources:
  HelloWorldAPI:
    Type: AWS::Serverless::Api
    Properties:
      Name: Hello World API
      StageName: Prod
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      PackageType: Image
      ImageConfig:
        Command: ["app.lambda_handler"]
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            RestApiId: !Ref HelloWorldAPI
            Path: /hello
            Method: get
    Metadata:
      Dockerfile: hello_world_docker
      DockerContext: .
      DockerTag: v1

We then create a docker file that creates our docker container.

FROM public.ecr.aws/lambda/python:3.9

COPY hello_world/. ./

RUN python3.9 -m pip install -r requirements.txt

CMD ["app.lambda_handler"]

This uses the Python 3.9 image from the Elastic Container Repository, then copies our code into the home directory of the container, runs python to install the requirements.txt (which just contains the requests module for now) and then sets the lambda_handler to be the entry point.

We then need to build the container with sam:

sam build --use-container -t template.yaml

You can then deploy the container to Lambda using:

sam deploy --stack-name sam-app --resolve-s3 --capabilities CAPABILITY_IAM --resolve-image-repos

(you may need to remove any stored config file from any previous sam build, likely to be called samconfig.toml or it may override, especially the s3 parameters).

This will deploy your lambda to a stack named sam-app, work out the s3 bucket to use for deployment, give permission to create IAM roles for deployment, and work out which ECR Repository to use.

With this deployed, you can go to the API Gateway URL mentioned in the output of the sam deploy command. You should see:

{"message": "hello world"}

Part 2 will create these commands inside a CodeBuild environment, allowing a Code Pipeline to take the code from a GitHub repository, then use the SAM CLI inside CodeBuild to build the container and deploy it automatically.

You can also use the aws cloudformation describe-stacks command inside CodeBuild to get the API generated, and use command line tools like sed to then replace this URL into other files, such as a front end (which could then also be deployed by CodeBuild, or another pipeline).

Part 2 coming soon.