As part of my learning curve on DynamoDB and its interaction with various AWS services, I am writing this article on how S3 event trigger triggers an action on a Lambda function to import CSV data into a DynamoDB table using AWS Management Console.

DynamoDB

DynamoDB is a key-value, non-relational database that uses a simple key-value method to store data. A key-value database stores data as a collection of key-value pairs in which a key serves as a unique identifier, which is called the Primary Key. Also known as Partition Key / Hash Key. Optionally it uses a Sort Key

DynamoDB is fully integrated with AWS Backups. You can use the DynamoDB console, API, and AWS Command Line Interface (AWS CLI) to enable automatic backups and restore for your DynamoDB tables.

What is Lambda?

AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or back-end service without provisioning or managing servers, capacity provisioning, automatic scaling, code monitoring and logging. You can trigger Lambda from over 200 AWS services and only pay for what you use.

S3 Bucket

Amazon Simple Storage (Amazon S3) is an object storage service. You can store and protect any amount of data for a range of use cases, like backup and restore, data lakes, archive, mobile application, and websites.

When someone creates an object or modifies (removes or updates) an object stored in your S3 bucket, S3 will trigger an event. You can use Lambda to process such event notifications from S3.

So add a trigger to our Amazon S3 bucket to call our Lambda function whenever new data arrives. Amazon S3 needs permission from the function's resource-based policy to invoke Lambda function.

An IAM Role

An IAM role is an AWS Identity. Every IAM role has its own permission policy that defines what that role can do and what it cannot do. It is like an IAM user without a password or an access key and a secret key. When you assume a role, it provides you with temporary security credentials for your role session. You use roles to allow the service to access resources in other services on your behalf. A role that a service assumes to perform actions on your behalf is called a service role. When a role serves a specialized purpose for a service, it is categorized as a service role for Lambda instances (for example), or a service-linked role.

Configuring the Role with proper policies/permissions is the key issue here just so that it can access proper AWS resources for sending out the message

Please visit my GitHub Repository for DynamoDB articles on various topics being updated on constant basis.

Let’s get started!

Objectives:

• Create a Amazon DynamoDB table. 

• Create an S3 bucket

• Upload a CSV file.

• All associated IAM roles needed for the solution, configured according to the principle of least privilege

• Creating Lambda Function with a timeout of 1 minute, which contains the code to import the CSV data into DynamoDB

• Test the CSV Data Import in Lambda

• Adding Event Triggers to the S3 Bucket to call our Lambda function whenever new data arrives.

• Test the setup - Testing S3 Event Trigger to Import New Data into DynamoDB

• Cleanup

Pre-requisites:

• AWS user account with admin access, not a root account.

• Cloud9 IDE with AWS CLI.

• Create an IAM role

Resources Used:

1. How to create a DynamoDB table
2. How to create a S3 bucket and a S3 Trigger
3. How to create a Lambda function
4. An IAM role with required permission policy

Steps for implementation to this project:

1. Create a Amazon DynamoDB table:

• On DynamoDB Dashboard / Tables / Under Create table , Table Details

Table Name: FriendsDDB
Partition key: Id
Type: String

Create Table

Status should be Active

2. Create an S3 bucket

• On Amazon S3 Console / Create bucket / Under Create bucket, General configuration

Bucket name: friends-s3

Create bucket

3. Upload a CSV file

• Click on the S3 bucket - friends-s3

File: friends.csv

• Under friends-s3, Objects, Upload / Under Upload, For Files and folders / Add files

select - friends.csv

Upload

4. All associated IAM roles needed for the solution, configured according to the principle of least privilege

• On the IAM dashboard / Roles / IAM dashboard / Create Role / Under Select trusted entity, Trusted entity type / Select AWS service / Under Use case / Select Lambda /

• Next

• Under Name, review, and create, Role details /

Role name: csv-lamda-role
Create role

• Click the role you just created csv-lambda-role
  drop-down Add permissions / select Attach policies

• Under Attach policy to csv-lambda-role / Search for AmazonDynamoDBFullAccess / Check the box / Attach poilicies

• Attach AmazonS3FullAccess in the same way.

5. Creating Lambda Function with a timeout of 1 minute, which contains the code to import the CSV data into DynamoDB

• On the Lambda Console / Functions / Create function / Select Author from scratch / Under Basic information

Function name: csv-s3-lambda
Runtime: From the drop-down choose Python 3.9

• Click on Change default execution role / select Use an existing Role

• Select the role you created just now - csv-lambda-role

Create function

• Once the function is created, it will open the main page of the Lambda function.

csv-lambda.py - contains Python code which uses boto3 APIs for AWS.

• In the csv-lambda.py -

  • Update table = dynamodb.Table("<DynamoDB Table") to
  • table = dynamodb.Table("FriendsDDB")

• The python code above does the following:

  • Imports the CSV file from S3 bucket.
  • Splits the CSV data into multiple strings.
  • Uploads data to the DynamoDB table.

import boto3
s3_client = boto3.client("s3")
dynamodb = boto3.resource("dynamodb")

table = dynamodb.Table("FriendsDDB")

def lambda_handler(event, context):
    bucket_name = event['Records'][0]['s3']['bucket']['name']
    s3_file_name = event['Records'][0]['s3']['object']['key']
    resp = s3_client.get_object(Bucket=bucket_name,Key=s3_file_name)
    data = resp['Body'].read().decode("utf-8")
    Students = data.split("\n")
    #print(friends)
    for friend in Friends:
        print(friend)
        friend_data = friend.split(",")
        # add to dynamodb
        try:
            table.put_item(
                Item = {
                    "id"        : friend_data[0],
                    "name"      : friend_data[1],
                    "Subject"   : friend_data[2]
                }
            )
        except Exception as e:
            print("End of file")

• Remove the existing code in the function code environment window.

• Copy and paste the code from the csv-lambda.py into the lambda_function.py under the Code Source.

• After updating the code, Click on Deploy button to save the code.

• Change the function timeout as follows:

  • Navigate to the Configuration 
  • click on General configuration / click on Edit 

• In the Edit Basic setting / change the Timeout value to 1 min

• Click on Save button

6. Test the CSV Data Import in Lambda:

• In the csv-s3-lambda lambda function page, click on the Test tab.

• Configure to test event data as follows:

Test event action: Create a new event

Name: Event name - csv

Template: Select Amazon S3 Put, Upon selection, it will be displayed as s3-put

• Below in the JSON code:

Under S3 → bucket → name → Enter friends-s3

arn": "arn:aws:s3:::friends-s3"

Under S3 → object → Key → Enter friends.csv

• Click on Create and then Save to save the changes.

Note: Make sure the S3 bucket name and file name are correct in the JSON.

• Click on Test in top-right Corner to trigger the lambda function.

• Once the lambda function is successfully executed, you will be able to see a detailed success message with table data.

• Go to the DynamoDB table and then select the FriendsDDB and click on Explore Table Items

7. Adding Event Triggers to the S3 Bucket to call our Lambda function whenever new data arrives.

• On the S3 Console / Click on the s3 bucket named friends-s3

• Click on the Properties tab / go down to Event notifications. 

• Click on Create event notification button

• Under General configuration /

Name: friends_upload

Suffix: .csv

All Object create events: check

Destination: Select Lambda Function

Lambda: Select csv-s3-lambda

Click on Save changes

• Now every time a CSV file is uploaded to our S3 bucket, it will trigger the lambda to import the CSV data into the DynamoDB table

8. Test the setup - Testing S3 Event Trigger to Import New Data into DynamoDB

file: friends1.csv

Upload the friends1.csv file to the friends-s3 bucket

• This upload event should triggered our Lambda function to import the CSV data into the DynamoDB table FriendsDDB.

• Go to the DynamoDB table FriendsDDB to see the changes.

• Click on the refresh button if items have not yet changed.

• You can see that new CSV data has been successfully imported into the DynamoDB table.

9. Cleanup

• delete the Lambda function
• delete the S3 bucket
• delete the DynamoDB table
• delete IAM Role

What we have done so far

• We have successfully created an Amazon DynamoDB Table.

• We have successfully created a Lambda function and configured it to import CSV data from S3 into DynamoDB.

• We have created an S3 event to trigger our Lambda function.

• We have tested the import of a new CSV file to the DynamoDB table.