Arpad Toth for AWS Community Builders

Posted on Apr 12 • Originally published at arpadt.com

Using API Gateway mapping templates for direct DynamoDB integrations

#aws #serverless #apigateway #dynamodb

Integrating API Gateway directly with DynamoDB can significantly enhance speed by reducing overall response time. Leveraging request and response mapping templates for various DynamoDB operations allows us to remove intermediary Lambda functions in the backend.

1. The scenario

I'm developing a weather application that periodically fetches current temperature data from the OpenWeather API. The application incorporates specific business logic to protect my plum, apple, and peach fruits from pests like plum moths, codling moths, and peach twig borers. It achieves this by monitoring growing degree units (GDUs) for each species and notifying me of the likely emergence day for larvae. Consequently, I can apply organic protection measures before they infest my fruits.

I store temperature and pest data in a DynamoDB table, adhering to the principles of the single table design. A REST-type API Gateway sits in front of the backend.

2. Eliminating transport functions

I've been exploring architectures that utilize native integrations between AWS services lately. In this context, I aim to avoid provisioning Lambda functions solely for data transmission, following the use Lambda to transform and not transport principle.

Consequently, I've opted not to implement Lambda functions between the API Gateway and DynamoDB. My objective is to construct this application with direct service integrations wherever feasible.

This adjustment has notably reduced existing endpoint response times to under 200ms on average, sometimes dipping below 100ms.

Additionally, utilizing direct integration translates to cost savings by bypassing Lambda function invocations. It also mitigates concerns related to managing Lambda function concurrencies.

3. Mapping templates

By removing the functions from the architecture, it's imperative to instruct API Gateway on processing client input data and configuring its behaviour during data retrieval from the database. This is accomplished using mapping templates.

Mapping templates are composed in the Apache Velocity Template Language (VTL) format. While initially challenging to work with, especially for developers accustomed to Lambda functions, they can provide granular control over data flow.

However, API Gateway's VTL support is not exhaustive. Successful local tests of a template do not guarantee identical processing by API Gateway. Consequently, ensuring proper template functionality may require additional effort.

4. TransactWriteItems operation from API Gateway

Let's examine the request mapping template for a POST endpoint.

In this scenario, I aim to add a new moth with various properties to the database. Because I'd like to retrieve all monitored moths via a GET /moths request later, I want to store their names in a dedicated item in the database.

To achieve this, I leverage DynamoDB's TransactWriteItems API.

4.1. Input

The client input data will be similar to the following:

{
  "newMoth": {
    "PK": "MOTH#Cydia pomonella",
    "SK": "MOTH#Cydia pomonella",
    "MothName": "Cydia pomonella",
    "BaseTemperature": "10",
    "EmergingGDU1": "104", // heat accumulation when larvae emerge from the eggs - 1st wave
    "MothEng": "Codling moth",
    // other properties
  },
  "toMothList": {
    "MothName": "Cydia pomonella"
  }
}

The newMoth object contains moth data, necessitating a PutItem operation. Simultaneously, the toMothList object specifies the moth's name for the UpdateItem operation, appending it to the MothsMonitored list in a dedicated singleton item.

4.2. TransactWriteItems request template

To execute both operations within a single database request without Lambda functions, we can create an integration request mapping template in API Gateway:

{
  "TransactItems": [
    {
      "Put": {
        "TableName": "Moths",
        "Item": {
          #set($inputRoot = $input.path('$.newMoth'))
          #foreach($key in $inputRoot.keySet())
              #set($value = $inputRoot.get($key))
              "$key": { "S": "$value" }#if($foreach.hasNext()),#end
          #end
        },
        "ConditionExpression": "attribute_not_exists(PK) AND attribute_not_exists(SK)"
      }
    },
    {
      "Update": {
        "TableName": "Moths",
        "Key": {
          "PK": { "S": "MOTHS" },
          "SK": { "S": "MOTHS" }
        },
        "UpdateExpression": "SET #attrName = list_append(if_not_exists(#attrName, :emptyList), :val)",
        "ExpressionAttributeNames": {
          "#attrName": "MothsMonitored"
        },
        "ExpressionAttributeValues": {
          ":val": { "L": [{ "S": "$input.path('$.toMothList.MothName')" }] },
          ":emptyList": { "L": [] }
        }
      }
    }
  ]
}

This template predominantly comprises typical DynamoDB PutItem and UpdateItem command codes.

However, let's emphasize a few key lines of the VTL part.

#set($inputRoot = $input.path('$.newMoth')) extracts the object from the input's newMoth property and stores it in the inputRoot variable. We then go over the keys using #foreach and wrap their values inside another object with the S DynamoDB data type.

The last thing in the Put part is that we must add commas after each item in the map, except the last one. API Gateway doesn't support trailing commas!

That's why the #if($foreach.hasNext()),#end part is in the code. hasNext returns true if there are more object properties left to iterate over, so in this case, we add the comma. When the logic processes the last item in the collection, hasNext will be false, and the conditional block's context (the ,) will not run.

The Update part only contains one VTL syntax: "$input.path('$.toMothList.MothName')". We access the second input object here and get the value for MothName. We add the moth name to the end of the dedicated MOTHS item's MothsMonitored list.

4.3. Permissions

Ensuring appropriate permissions is crucial. API Gateway must be granted permission to execute PutItem and UpdateItem actions on the designated DynamoDB table.

5. GetItem

For the GET /moth?mothName=<MOTHNAME> endpoint, I've created an integration response mapping template in API Gateway to transform DynamoDB's moth data into the desired client format.

Since a single item is fetched, a GetItem operation is performed on DynamoDB, necessitating the API Gateway role having a GetItem permission.

5.1. Request template

A straightforward request template is crafted to add the string data type descriptor to the DynamoDB payload:

{
"TableName": "Moths",
  "Key": {
    "PK": { "S": "MOTH#$input.params('mothName')" },
    "SK": { "S": "MOTH#$input.params('mothName')" }
  }
}

The moth name, extracted from query parameters, will be part of the primary and sort keys.

5.2. Response template

DynamoDB responds to the GetItem operation with a payload similar to this:

{
  "Item": {
    "PK": {
      "S": "Value1"
    },
    "SK": {
      "S": "Value2"
    },
    "Attribute1": {
      "S": "Value1"
    },
    "Attribute2": {
      "S": "Value2"
    },
    // other attributes
  },
  "ConsumedCapacity": {
    // ...
  }
}

However, the desired client format slightly differs from the database output as I want it to be an object like this:

{
  "Attribute1": "Value1",
  "Attribute2": "Value2",
  // other attributes
}

So I want a flat object without any Item or other wrappers. I also remove the S data type descriptor from the database response. Thirdly, I don't want to return PK and SK or any other local or global secondary index attributes to the client. I only need data attributes in the client. Lastly, I'll return a short message if the requested item is not saved to the database.

To provide the response the client needs, I created the following transformation:

#set($inputRoot = $input.path('$'))
#if($inputRoot.Item && !$inputRoot.Item.isEmpty())
  #set($excludeKeys = ["PK", "SK"])
  #set($resultMap = {})
  #foreach($key in $inputRoot.Item.keySet())
    #if(!$excludeKeys.contains($key))
      #set($dummy = $resultMap.put($key, $inputRoot.Item.get($key).S))
    #end
  #end
  {
    #foreach($key in $resultMap.keySet())
      "$key": "$resultMap.get($key)"
      #if($foreach.hasNext()),#end
    #end
  }
#else
  {"messsage": "No item found"}
#end

We create a temporary map called resultMap that stores the item's key/value pairs without the S data type descriptor. The #if(!$excludeKeys.contains($key)) condition checks if the key is not an index attribute, i.e., not PK or SK. If it's not, we store the key/value pair in resultMap. This way, we can remove PK and SK, which don't carry data, from the database response object.

The next challenge is to create a JSON object with no comma after the last property. We can achieve it by creating a second #foreach block to iterate over resultMap's keys. Then, we apply the same condition technique to add commas everywhere except after the last element.

That's it! We now have two fast endpoints, a write and a read one, which work without provisioning and invoking any Lambda functions!

6. Considerations

Handling different data types (e.g., numbers, lists, booleans) may necessitate more intricate VTL code. Here, I've opted to represent everything as strings to simplify VTL composition.

While writing VTL may initially be time-consuming compared to traditional programming languages, the benefits of expedited endpoints, reduced management overhead, and cost savings make it a compelling choice.

7. Summary

Implementing direct integrations between API Gateway and DynamoDB offers expedited performance, decreased costs, and less management overhead. Utilizing VTL syntax enables seamless mapping of payloads to and from the database.