AWS DynamoDB: Comparing Lambda & Step Functions To Scan 1MB Data

while working In a project i was exploring the Lambda, DynamoDB, and step function, while my aim was to fetch data from DDB in both the Lambda and in Step Function, which led me to an idea of comparing both the method with time, memory usage, and cost for each method. to this idea i added another layer of scanning 1MB of data from DynamoDB. now without wasting time let's explore the setup, execution and cost which should be considered.

Setup

To test both the scenario, let's setup required Lambda & step function, for both method will be sharing code to scan DB records, all the creation of Lambda & step function definition will be done in the aws console.

stepFunction Definition:

{
  "Comment": "Scan all the DB records",
  "StartAt": "Scan User Data",
  "States": {
    "Scan User Data": {
      "Type": "Task",
      "Parameters": {
        "TableName": "userInfo",
        "Limit": 50
      },
      "Resource": "arn:aws:states:::aws-sdk:dynamodb:scan",
      "End": true
    }
  }
}

Lambda Logic: Below code aims to scan the data from DDB, in the starting of the process i noted the time and and the completion of the task i again noted the time by the end i can find the difference from start to end, to find the total time taken to complete the task.

const { DynamoDBClient, ScanCommand } = require("@aws-sdk/client-dynamodb");

const dynamoDB = new DynamoDBClient({ region: 'us-east-1' });

const scanWholeTable = async (startKey) => {
    let items = [];
    const startTime = new Date().getTime();
    try {
      const params = {
        TableName: 'userInfo',
        ExclusiveStartKey: startKey,
        Limit: 25,
      };

      // prepare the params to scan the table
      const command = new ScanCommand(params);
      const data = await dynamoDB.send(command);

      items = items.concat(data.Items);

      // If there are more items to scan, recursively call the scanDynamoDBTable function with the last evaluated key
      if (data.LastEvaluatedKey) {
        return scanWholeTable(data.LastEvaluatedKey);
      }

      // check the time taken to complete the scan
      const endTime = new Date().getTime();
      const totalTime = endTime - startTime
      console.log('[INFO] time taken to complete the db scan', totalTime)
      return items;
    } catch (error) {
      console.log('[ERROR] Error while scanning the table', error);
      throw error;
    }
}

module.exports.handler = async (event, context) => {
  try {
      const finalUserInfo = await scanWholeTable(null);
      console.log('[INFO] UserInfo', finalUserInfo);
      return {
        statusCode: 200,
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({userData:finalUserInfo, length: finalUserInfo.length}),
      }
    } catch (error) {
      console.info('Error in the handler:', error);
      return error;
    }
};

Time to complete the task.

Once all the setup is done, I triggered both the scenario multiple time to determine amount of time both method takes to scan 1MB of data from DDB.

Lambda:

• I aimed trigger the lambda multiple times.
• As I triggered multiple time, Lambda execution time ranged between 500-1100 milliseconds for scanning 1MB of data.
• To complete the task Lambda execution took 1-2 seconds
• And for The first trigger Lambda took 1253 milliseconds.

• As triggering multiple time lambda showed reduced time of 544 milliseconds.

Step Function:

• Continuously triggering the step function surprisingly took only 266 milliseconds to complete the task.
• Task initialisation and completion took 400-500 milliseconds.
• Apart from some lag, both Lambda and Step Function times are worth considering.

Cost Consideration.

Let's use the AWS Price Calculator to estimate the cost of scanning 1MB on DynamoDB.

Lambda Cost:
Let's configure the region (us-east-1), architecture (x84), memory (512), timeout (30), and trigger per day (let's keep it at 50).once all the configuration is set let's break down the cost.

• For 50 requests per day, which total will be 1,520 per month, and memory usage of 608.33 GB for computing.
• AWS provides 400,000 GB as a free tier for compute, resulting in a compute cost of 0 USD.
• Currently AWS Lambda makes 1 million triggers as part of the free tier, which result in our process to cost 0 USD.
• Running the Lambda will cost 0 USD until it reaches 400k compute and 1 million requests.

Step Function Cost:
Now configure the region (us-east-1), select standard workflow, and set the workflow requests to 50 per day, 5 state transition of the step function, let's consider there will be 5 states after it to transform data and send back the result.

• With 1,520 requests triggered to the Step Function per month.
• AWS provides 4,000 transitions as a free tier for the step function service, and for our step function it will be total of 7,604 transitions per month.
• As a result above point the billable transitions amount will be 3,604 transitions after 7,604 - 4,000.
• The cost for one transition state is 0.000025 USD.
• leading to an estimated cost of 0.09 USD.

Summary

As i delve into comparing the Lambda & Step Function to scan 1MB of data from DynamoDb, yield insightful observations, on the time & cost for both methods, where Lambda free-tier benefits overshadow the Step Function consistent performance.