DEV Community

Matt Martz
Matt Martz

Posted on

DynamoDB, Lambdas and API GW with localstack (or not)

#aws #cdk #localstack

My original goal for this post was to get a local version of DynamoDB, API Gateway and Lambdas up and running using localstack. Unfortunately localstack doesn't seem to play well with cloudformation + lambdas... At a certain point this tutorial will transition from a localstack to a AWS-native approach. I'm keeping the localstack related commentary because I think it's useful to people looking to do the same.

My goal for this post is to get DynamoDB, API Gateway and Lambdas up and running locally on AWS using cloudformation).

This is a very small expansion of this post:

elthrasher

Exploring AWS CDK - Loading DynamoDB with Custom Resources

Matt Morgan ・ Jan 6 '20

#aws #devops #typescript #serverless

I had intended for it to be heavier on the local dev experience but that didn't pan out.

Alt Text

Table of Contents

Getting Started

Alt Text

The code for this is here:

GitHub logo martzcodes / blog-cdk-localstack

Make sure you have cdk installed

Locally, I have this installed:

$ node --version
v12.13.1
$ npm --version
6.12.1
$ aws --version
aws-cli/2.0.23 Python/3.7.4 Darwin/19.5.0 botocore/2.0.0dev27
$ cdk --version
1.45.0 (build 0cfab15)
Enter fullscreen mode Exit fullscreen mode

To initialize a project:

$ cdk init --language typescript
Applying project template app for typescript
Initializing a new git repository...
Executing npm install...

# Welcome to your CDK TypeScript project!

This is a blank project for TypeScript development with CDK.

The `cdk.json` file tells the CDK Toolkit how to execute your app.

## Useful commands

 * `npm run build`   compile typescript to js
 * `npm run watch`   watch for changes and compile
 * `npm run test`    perform the jest unit tests
 * `cdk deploy`      deploy this stack to your default AWS account/region
 * `cdk diff`        compare deployed stack with current state
 * `cdk synth`       emits the synthesized CloudFormation template
Enter fullscreen mode Exit fullscreen mode

There are some dependencies that will need to get installed... let's just take care of all of that now. Run: npm i @aws-cdk/aws-apigateway @aws-cdk/aws-dynamodb @aws-cdk/aws-iam @aws-cdk/aws-lambda @aws-cdk/aws-s3 @aws-cdk/aws-s3-assets @aws-cdk/aws-lambda-nodejs --save. S3 is needed because that's where the lambda code will actually live. Ends up... probably could've skipped the S3 dependencies, but they don't hurt anything...

Creating the DynamoDB table

Now, let's start to get our feet wet with CDK... Time to create the DynamoDB table. For this project we're going to make a table that tracks a user's ratings of beer. In my lib/blog-cdk-localstack-stack.ts file I'm going to update the code to:

import { AttributeType, Table, BillingMode } from "@aws-cdk/aws-dynamodb";
import { Construct, RemovalPolicy, Stack, StackProps } from "@aws-cdk/core";

export class BlogCdkLocalstackStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const modelName = "Beer";

    const dynamoTable = new Table(this, modelName, {
      billingMode: BillingMode.PAY_PER_REQUEST,
      partitionKey: {
        name: `UserId`,
        type: AttributeType.STRING,
      },
      sortKey: {
        name: `${modelName}Id`,
        type: AttributeType.STRING,
      },
      removalPolicy: RemovalPolicy.DESTROY,
      tableName: modelName,
    });
  }
}
Enter fullscreen mode Exit fullscreen mode

This creates the dynamoDB table with a partition key as the UserId and a SortKey with the BeerId. Since this is a user-focused app this will enable me to get all of a user's beer ratings by using the partition key, and if I want to filter down to a specific beer and its ratings I can do that (there's a LOT more you can do with partition / sort keys but it's not needed for this example).

Alt Text

So far so good...

DynamoDB + localstack

Next, I'll run cdk synth BlogCdkLocalstackStack --profile personal > BlogCdkLocalstackStack.template.yml which will output the CloudFormation template to create the DynamoDB table. Now I need to get localstack somewhat set up...

I'll start with a somewhat standard docker-compose file:

version: "3.8"
services:
  localstack:
    image: localstack/localstack
    ports:
      - "4566-4599:4566-4599"
      - "8080:8080"
    environment:
      - SERVICES=s3,lambda,cloudformation,apigateway,dynamodb
      - DEBUG=true
      - DATA_DIR=/tmp/localstack/data
      - LAMBDA_EXECUTOR=local
      - LAMBDA_REMOTE_DOCKER=false
      - DOCKER_HOST=unix:///var/run/docker.sock
      - START_WEB=1
    volumes:
      - ./tmp:/tmp/localstack
      - "/var/run/docker.sock:/var/run/docker.sock"
Enter fullscreen mode Exit fullscreen mode

And run docker-compose up -d. After waiting (and periodically checking the logs) to make sure it's up and ready... I can now run:

aws cloudformation create-stack --stack-name BlogCdkLocalstackStack --template-body file://./BlogCdkLocalstackStack.template.yml --endpoint-url http://localhost:4566 which should initialize the DynamoDB table in localstack. If I check the logs I get some log statements that don't necessarily give me a warm and fuzzy:

localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Currently processing stack resource BlogCdkLocalstackStack/BeerE09E1431: None
localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Deploying CloudFormation resource (update=False, exists=False, updateable=False): {'Type': 'AWS::DynamoDB::Table', 'Properties': {'KeySchema': [{'AttributeName': 'UserId', 'KeyType': 'HASH'}, {'AttributeName': 'BeerId', 'KeyType': 'RANGE'}], 'AttributeDefinitions': [{'AttributeName': 'UserId', 'AttributeType': 'S'}, {'AttributeName': 'BeerId', 'AttributeType': 'S'}], 'BillingMode': 'PAY_PER_REQUEST', 'TableName': 'Beer'}, 'UpdateReplacePolicy': 'Delete', 'DeletionPolicy': 'Delete', 'Metadata': {'aws:cdk:path': 'BlogCdkLocalstackStack/Beer/Resource'}}
localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.utils.cloudformation.template_deployer: Running action "create" for resource type "DynamoDB::Table" id "BeerE09E1431"
localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.utils.cloudformation.template_deployer: Request for resource type "DynamoDB::Table" in region us-east-1: create_table {'TableName': 'Beer', 'AttributeDefinitions': [{'AttributeName': 'UserId', 'AttributeType': 'S'}, {'AttributeName': 'BeerId', 'AttributeType': 'S'}], 'KeySchema': [{'AttributeName': 'UserId', 'KeyType': 'HASH'}, {'AttributeName': 'BeerId', 'KeyType': 'RANGE'}], 'ProvisionedThroughput': {'ReadCapacityUnits': 5, 'WriteCapacityUnits': 5}}
localstack_1  | 2020-06-18T14:39:51:WARNING:localstack.services.cloudformation.cloudformation_starter: Unable to determine physical_resource_id for resource <class 'moto.dynamodb2.models.Table'>
localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Currently processing stack resource BlogCdkLocalstackStack/CDKMetadata: None
localstack_1  | 2020-06-18T14:39:51:WARNING:moto: No Moto CloudFormation support for AWS::CDK::Metadata
localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Running CloudFormation stack deployment loop iteration 1
Enter fullscreen mode Exit fullscreen mode

But if I run aws dynamodb list-tables --endpoint-url http://localhost:4566 the table is definitely listed. From here I can verify the stacks: aws cloudformation list-stacks --endpoint-url http://localhost:4566, delete the stack aws cloudformation delete-stack --stack-name BlogCdkLocalstackStack --endpoint-url http://localhost:4566 and attempt to verify that the delete happened... turns out deleting the stack did not delete the table... could be a quirk of localstack or I'm using the wrong command...

Alt Text

That kinda works...

Creating the Lambdas

Ultimately, I'm going to want to have two endpoints... GET all user's and their beer ratings... and Update a User's beer ratings. Clearly we'd want some more fidelity there, but this is a simple demo.

Let's start with the update lambda. I'll be somewhat basing my code off of this example: https://github.com/aws-samples/aws-cdk-examples/tree/master/typescript/api-cors-lambda-crud-dynamodb but I want to use the @aws-cdk/aws-lambda-nodejs cdk module so my lambda can be stored as a typescript file.

import { DynamoDB } from "aws-sdk";

const db = new DynamoDB.DocumentClient();

export const handler = async (event: any): Promise<any> => {
  if (!event.body) {
    return {
      statusCode: 400,
      body: "invalid request, you are missing the parameter body",
    };
  }

  const body = JSON.parse(event.body);

  try {
    const review = await db
      .update({
        TableName: 'Beer',
        Key: {
          UserId: event.pathParameters.user,
          BeerId: event.pathParameters.beer,
        },
        UpdateExpression: "set rating = :r",
        ExpressionAttributeValues: {
          ":r": `${body.rating || 0}`
        },
        ReturnValues: "ALL_NEW",
      })
      .promise();
    console.log(`Update complete. ${JSON.stringify(review)}`);
    return {
      statusCode: 200,
      headers: {},
      body: JSON.stringify(review),
    };
  } catch (e) {
    console.error("GET failed! ", e);
    return {
      statusCode: 400,
      headers: {},
      body: `Update failed: ${e}`,
    };
  }
};
Enter fullscreen mode Exit fullscreen mode

The GET ALL lambda ends up being very similar except it uses a scan of the DB which 99% of the time you should absolutely NOT do since it actively reads every item in your table!

Next... the lambdas need to get tied into the main blog-cdk-localstack-stack.ts file. Where we left off we only had the table. After creating the table we need to create the lambdas. For that I'm using the experimental NodejsFunction which can directly compile typescript files (it's quite convenient, but could be minimized more).

const updateReview = new NodejsFunction(this, "updateReviewFunction", {
      entry: `${lambdaPath}/update-review.ts`,
      handler: "handler",
      runtime: Runtime.NODEJS_12_X,
    });

    const getReviews = new NodejsFunction(this, "readReviewFunction", {
      entry: `${lambdaPath}/get-reviews.ts`,
      handler: "handler",
      runtime: Runtime.NODEJS_12_X,
    });
Enter fullscreen mode Exit fullscreen mode

The lambdas also need to have access to dynamodb:

    dynamoTable.grantReadWriteData(updateReview);
    dynamoTable.grantReadData(getReviews);
Enter fullscreen mode Exit fullscreen mode

Time to deploy this to localstack and invoke the functions manually...

Alt Text

This should go well, right?

Lambdas + localstack via cloudformation

At this point I figured out CloudFormation lambdas on localstack STILL do not work

I tried a number of things to no avail. The error has to do with where localstack / cloudformation is trying to store the lambda code. Lambda code is stored on S3... localstack had s3 turned on but the parameter was supposedly missing. This is the error for anyone curious:

localstack_1  | 2020-06-18T16:26:12:DEBUG:localstack.services.cloudformation.cloudformation_listener: Error response for CloudFormation action "CreateStack" (400) POST /: b'<ErrorResponse xmlns="http://cloudformation.amazonaws.com/doc/2010-05-15/">\n  <Error>\n    <Type>Sender</Type>\n    <Code>Missing Parameter</Code>\n    <Message>Missing parameter AssetParameters7708d9acea8c2c3d47875fb4c6499e05f52897d0ff284eb6811ad1fc7d97eeadS3Bucket9261AF10</Message>\n  </Error>\n  <RequestId>cf4c737e-5ae2-11e4-a7c9-ad44eEXAMPLE</RequestId>\n</ErrorResponse>'
Enter fullscreen mode Exit fullscreen mode

So, I'm going to skip localstack, for now...

Alt Text

API Gateway

Finally, we need to set up the api gateway to use the lambdas:

    const api = new LambdaRestApi(this, 'beer-api',
    {
      handler: getReviews,
      proxy: false
    });

    const users = api.root.addResource('users');
    users.addMethod("GET", new LambdaIntegration(getReviews));
    const user = users.addResource('{user}');
    const userReview = user.addResource('{beer}');
    userReview.addMethod("POST", new LambdaIntegration(updateReview));
Enter fullscreen mode Exit fullscreen mode

First we create a lambda-based REST api. It requires a default handler but with proxy set to false nothing will actually go to it... they need to be manually defined which is what the following lines do.

Deploying to AWS

Since localstack is out of the picture, to get this deployed to AWS I'll need to do cdk bootstrap --profile personal first... Which will initialize the stack on my AWS account. If I follow that by cdk deploy --profile personal it will ask if I want to deploy the changeset (I do). After a minute or two of logs I get the result:

Outputs:
BlogCdkLocalstackStack.beerapiEndpointE86AAAC2 = https://1ruxq80qo2.execute-api.us-east-1.amazonaws.com/prod/
Enter fullscreen mode Exit fullscreen mode

Which, if I add add my api routes I can hit https://1ruxq80qo2.execute-api.us-east-1.amazonaws.com/prod/users to get a 200 response with no items and alternately do a POST to https://1ruxq80qo2.execute-api.us-east-1.amazonaws.com/prod/users/123/123 with the body {"rating": 8} which responds with the updated item:

{
    "Attributes": {
        "BeerId": "123",
        "UserId": "123",
        "rating": "8"
    }
}
Enter fullscreen mode Exit fullscreen mode

It's not local, but it WAS easy...

Alt Text

Clean-up

To remove everything in AWS it's as simple as running cdk destroy --profile personal

$ cdk destroy --profile personal
Are you sure you want to delete: BlogCdkLocalstackStack (y/n)? y
BlogCdkLocalstackStack: destroying...
   0 | 10:12:39 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::Stack  | BlogCdkLocalstackStack User Initiated
   0 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Account    | beer-api/Account (beerapiAccount2152DC9C) 
   0 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::CDK::Metadata          | CDKMetadata 
   0 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Permission     | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer86A593EA) 
   0 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Permission     | beerapiusersuserGETApiPermissionBlogCdkLocalstackStackbeerapi01021E61GETusersuser199DA8EE 
   0 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Permission     | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.Test.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer16885B45) 
   1 | 10:12:42 PM | DELETE_COMPLETE      | AWS::ApiGateway::Account    | beer-api/Account (beerapiAccount2152DC9C) 
   1 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Permission     | beerapiusersuserGETApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61GETusersuser9933087C 
   1 | 10:12:43 PM | DELETE_IN_PROGRESS   | AWS::IAM::Role              | beer-api/CloudWatchRole (beerapiCloudWatchRole81F996BB) 
   2 | 10:12:43 PM | DELETE_COMPLETE      | AWS::CDK::Metadata          | CDKMetadata 
   3 | 10:12:44 PM | DELETE_COMPLETE      | AWS::IAM::Role              | beer-api/CloudWatchRole (beerapiCloudWatchRole81F996BB) 
   4 | 10:12:52 PM | DELETE_COMPLETE      | AWS::Lambda::Permission     | beerapiusersuserGETApiPermissionBlogCdkLocalstackStackbeerapi01021E61GETusersuser199DA8EE 
   5 | 10:12:52 PM | DELETE_COMPLETE      | AWS::Lambda::Permission     | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.Test.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer16885B45) 
   6 | 10:12:52 PM | DELETE_COMPLETE      | AWS::Lambda::Permission     | beerapiusersuserGETApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61GETusersuser9933087C 
   7 | 10:12:52 PM | DELETE_COMPLETE      | AWS::Lambda::Permission     | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer86A593EA) 
   7 | 10:12:53 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Stage      | beer-api/DeploymentStage.prod (beerapiDeploymentStageprod3F2FBBD3) 
   8 | 10:12:54 PM | DELETE_COMPLETE      | AWS::ApiGateway::Stage      | beer-api/DeploymentStage.prod (beerapiDeploymentStageprod3F2FBBD3) 
   8 | 10:12:55 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Deployment | beerapiDeployment3077C14B26da823256c4ecc7e0f6cf539683f3ef 
   9 | 10:12:56 PM | DELETE_COMPLETE      | AWS::ApiGateway::Deployment | beerapiDeployment3077C14B26da823256c4ecc7e0f6cf539683f3ef 
   9 | 10:12:57 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Method     | beerapiusersuserGETCE81253D 
   9 | 10:12:57 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Method     | beer-api/Default/users/{user}/{beer}/POST (beerapiusersuserbeerPOST2D1FBB1F) 
  10 | 10:12:57 PM | DELETE_COMPLETE      | AWS::ApiGateway::Method     | beerapiusersuserGETCE81253D 
  11 | 10:12:58 PM | DELETE_COMPLETE      | AWS::ApiGateway::Method     | beer-api/Default/users/{user}/{beer}/POST (beerapiusersuserbeerPOST2D1FBB1F) 
  11 | 10:12:58 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Function       | readReviewFunction (readReviewFunction94EC729D) 
  11 | 10:12:58 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Function       | updateReviewFunction (updateReviewFunctionF0FF17E0) 
  11 | 10:12:58 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Resource   | beer-api/Default/users/{user}/{beer} (beerapiusersuserbeerE37FAAD9) 
  12 | 10:12:59 PM | DELETE_COMPLETE      | AWS::Lambda::Function       | readReviewFunction (readReviewFunction94EC729D) 
  13 | 10:12:59 PM | DELETE_COMPLETE      | AWS::ApiGateway::Resource   | beer-api/Default/users/{user}/{beer} (beerapiusersuserbeerE37FAAD9) 
  14 | 10:12:59 PM | DELETE_COMPLETE      | AWS::Lambda::Function       | updateReviewFunction (updateReviewFunctionF0FF17E0) 
  14 | 10:13:00 PM | DELETE_IN_PROGRESS   | AWS::IAM::Policy            | readReviewFunction/ServiceRole/DefaultPolicy (readReviewFunctionServiceRoleDefaultPolicy2543AE62) 
  14 | 10:13:00 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Resource   | beer-api/Default/users/{user} (beerapiusersuser40A97AB3) 
  14 | 10:13:00 PM | DELETE_IN_PROGRESS   | AWS::IAM::Policy            | updateReviewFunction/ServiceRole/DefaultPolicy (updateReviewFunctionServiceRoleDefaultPolicyD907B783) 
  15 | 10:13:00 PM | DELETE_COMPLETE      | AWS::IAM::Policy            | readReviewFunction/ServiceRole/DefaultPolicy (readReviewFunctionServiceRoleDefaultPolicy2543AE62) 
  16 | 10:13:00 PM | DELETE_COMPLETE      | AWS::ApiGateway::Resource   | beer-api/Default/users/{user} (beerapiusersuser40A97AB3) 
  17 | 10:13:01 PM | DELETE_COMPLETE      | AWS::IAM::Policy            | updateReviewFunction/ServiceRole/DefaultPolicy (updateReviewFunctionServiceRoleDefaultPolicyD907B783) 
  17 | 10:13:01 PM | DELETE_IN_PROGRESS   | AWS::IAM::Role              | readReviewFunction/ServiceRole (readReviewFunctionServiceRole77A9B597) 
  17 | 10:13:01 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Resource   | beer-api/Default/users (beerapiusers7623AC8A) 
  17 | 10:13:02 PM | DELETE_IN_PROGRESS   | AWS::DynamoDB::Table        | Beer (BeerE09E1431) 
  17 | 10:13:02 PM | DELETE_IN_PROGRESS   | AWS::IAM::Role              | updateReviewFunction/ServiceRole (updateReviewFunctionServiceRole57FA81FA) 
  18 | 10:13:02 PM | DELETE_COMPLETE      | AWS::ApiGateway::Resource   | beer-api/Default/users (beerapiusers7623AC8A) 
  19 | 10:13:03 PM | DELETE_COMPLETE      | AWS::IAM::Role              | readReviewFunction/ServiceRole (readReviewFunctionServiceRole77A9B597) 
  19 | 10:13:04 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::RestApi    | beer-api (beerapi67AE41CF) 
  20 | 10:13:04 PM | DELETE_COMPLETE      | AWS::IAM::Role              | updateReviewFunction/ServiceRole (updateReviewFunctionServiceRole57FA81FA) 
  21 | 10:13:04 PM | DELETE_COMPLETE      | AWS::ApiGateway::RestApi    | beer-api (beerapi67AE41CF) 

 ✅  BlogCdkLocalstackStack: destroyed
Enter fullscreen mode Exit fullscreen mode

Final thoughts...

It's definitely possible to run everything in localstack by manually running all the commands... but that ultimately defeats the purpose of using CDK.

Do you have any tips for doing local development across API Gateway -> Lambdas -> DynamoDB? With or without localstack?

Alt Text

Top comments (2)

Collapse
 
jrothman profile image
Joel Rothman •

Thanks Matt - I'm also keen to get a "full-stack" local dev environment going for this use case. Hopefully someone can enlighten us, otherwise please update the post if you do figure it out!

Collapse
 
jleck profile image
James Leckenby • • Edited

Hi Matt,

I've been investigating this, and I believe the lambda issue is actually due to the way CDK works, rather than an issue with localstack.

When you create a lambda through CDK, it uses the CDK bucket to upload the code (this bucket is created when you run "cdk bootstrap").

The parameters are in fact missing because CDK would normally pass them when deploying (e.g. the parameter for bucket would point to the cdk staging bucket in S3).

See the following for passing parameters when calling cloudformation:
aws.amazon.com/blogs/devops/passin...

We should be able to get this working if we manually upload the code (as CDK would do), then set the parameters to match.

Read next

notharshhaa profile image

100 AWS DevOps Interview Questions and Answers

H A R S H H A A -

gabrielkoo profile image

I bought us-east-1.com: A Look at Security, DNS Traffic, and Protecting AWS Users

Gabriel Koo -

afiqiqmal profile image

AWS Session Manager vs SSH

Hafiq Iqmal -

abidemi profile image

Creating an Instance on AWS using Ubuntu - AWS Tutorial

Akinsooto Abidemi -