Using SAM to deploy an node.js lambda to consume Kinesis stream

If you only need inline code for the lambda, the cloudformation should be enough. But if you need to do npm install for your lambda, using SAM would be more suitable and I will show you the steps in the next.

Prerequisite

• Install SAM CLI (SAM official install page)
• Create a S3 bucket first for storing the lambda

General Plan

1. Write a sam.yaml file describing a lambda to consume a Kinesis stream
2. Let the sam.yaml able to invoke a Makefile command to do npm install
3. Use sam build to do the preparation works before deployment
4. Use same deploy to upload lambda function into the S3 and setup all the other resources that needed

Details

- sam/
  - sam.yaml
- src/
  - index.js
  - package.json
- Makefile

Step 1: writing saml.yaml

• Setup a new lambda
• Let the lambda to consume the Kinese stream (SAM will help you to handle role and policy issues automatically)
• Tell sam build to make use of makefile for the building

sam/sam.yaml

Transform: 'AWS::Serverless-2016-10-31'

Resources:
  MyLambdaName:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: {YOUR_LAMBDA_FUNCTION_NAME}
      Description: {YOUR_LAMBDA_DESCRIPTION}
      Handler: index.handler
      Runtime: nodejs14.x
      CodeUri: ../  # Because Makefile in '../'
      MemorySize: 128
      Timeout: 10
      Events:
        KinesisStream:
          Type: Kinesis
          Properties:
            BatchSize: 1 # Modify this for batch consuming
            Enabled: true
            Stream: {YOUR_KINESIS_STREAM_ARN}
            MaximumBatchingWindowInSeconds: 0
            StartingPosition: LATEST
    Metadata:
      BuildMethod: makefile

Step 2: writing Makefile

• Because in the sam file, we named our lambda resource as MyLambdaName, so we add a command named build-MyLambdaName in the makefile and it will be invoked automatically by running sam build
• What the make command do is coping the source file into the tmp folder which created by the sam build and running npm install inside the tmp folder to generate the node_modules

Makefile

build-MyLambdaName:
  @echo "Packaging the source file..."
  @cp -R src/* $(ARTIFACTS_DIR)
  @npm --prefix $(ARTIFACTS_DIR) i --production

Step 3: running sam build

• My sam file is in the sam/ folder, so I need to specify the file position with the parameter -t
• After running the build command, a new folder .aws-sam/ will be created, sam deploy will auto-search and use this folder (don't forget put .aws-sam into the .gitignore)

sam build -t sam/sam.yaml

Step 4: running sam deploy

• sam deploy will make use of the .aws_sam folder which created by the sam build
• Deployment includes uploading lambda function into S3 bucket and setup all the resources in the sam.yaml

sam deploy 
    --region {AWS_REGION} \
    --stack-name {STACK_NAME_IN_CLOUDFORMATION} \
    --s3-bucket {S3_BUCKET_NAME} \
    --s3-prefix {S3 FOLDER NAME IN THE BUCKET}

End

Later when you do any modification to the lambda, you only need to re-run sam build and same deploy.

It will create and upload a new file into your S3 and update lambda function to use the latest file.