Creating Lambda backed Custom Cloudformation resource in C#

Notification 1 : All code related to this blog post can be found here  https://github.com/turjachaudhuri/CF-custom-resources

So , what is this about?

So , while doing a personal proof-of-concept I had to setup a CloudFormation template for a DynamoDB . The table also needed some initial values for config purposes , like master data that I needed to load into that DynamoDB table . Though there were only a few values , I still wanted to automate the process to some extent so that the master data will be setup as soon as the table is created .

This would help a lot , mostly during migrations as this will reduce one manual step in the process . Also , this can be extended into any number of use cases as need be , which will become evident by the end of this blog post.

Can we simply not use CloudFormation?

CloudFormation(CF) is awesome for creating AWS resources in a maintainable , consistent way . However , CF is limited in a sense that it only supports a subset of all the AWS resources and only a few specific operations .

Say , you want to trigger a separate event / call an external API when a resource within CloudFormation gets created. Currently , CF does not support this . Or say , you want to create an AWS resource which is not yet supported by CloudFormation. Or say , you want to load some master data/config data into a DynamoDB table as soon as the DynamoDB table gets created . CloudFormation does not support any of these operations out of box ,as of now.

Custom Resources to the rescue !!

"Custom resources enable you to write custom provisioning logic in templates that AWS CloudFormation runs anytime you create, update (if you change the custom resource), or delete stacks"- AWS Official post

Basically , in case of a lambda backed custom resource  , what you can do is as follows -

Specify a custom resource in the CloudFormation template.
Link a lambda function to the custom resource . When the CloudFormation stack gets created/updated/deleted , CF sends a request to this lambda function.

The lambda function contains the actual code to do what is needed , which can be loading a DynamoDB table from master data , call an API endpoint to create an AWS/non-AWS resource.

So , custom resources are basically an extension of CF. However , there are a lot of things to keep in mind while designing custom resources.

Let's get started

A custom resource is defined in the CloudFormation template as -

  "PopulateMasterData": {
    "Type": "Custom::PopulateMasterData",
    "Properties": {
      "ServiceToken": {
        "Fn::GetAtt": [
                  "CustomResourceFunction",
                  "Arn"
                      ]
                      },
      "TableName": {
        "Fn::Sub": "${DynamoDBTableName1}"
      }
    }
  }

The only required property is **ServiceToken **which refers to the ARN of the lambda function that is to be invoked by CloudFormation while creating the custom resource.

Within the properties section of the above JSON , any property other than ServiceToken  is passed into lambda function as an event by CloudFormation during the custom resource create/update/delete.

Basically , in the properties section you need to mention all the parameters that will define the custom resource you can use . For example , in my case, the lambda function basically needs to push some configuration data into a DynamoDB created within the same stack .

So , I have passed the tableName  as a property . In the lambda function I reference this tableName property and use it to push values into that DynamoDB table. If you are creating an AWS resource which is not supported by CloudFormation , then you might send the properties of the resource you want to create , and then reference those properties in the lambda function to create the actual resource via an API call , and so on.

Things to remember

There are a few things which are a must to keep in mind while designing the associated lambda function that CloudFormation calls during the creation/updation/deletion of your custom resource. These are mentioned in detail here -

https://aws.amazon.com/premiumsupport/knowledge-center/best-practices-custom-cf-lambda ,and I will mention how I have tried to accommodate them in my design.

1 . Build your custom resources to report, log, and handle failure gracefully

"Exceptions can cause your function code to exit without sending a response. Because CloudFormation requires an HTTPS response to confirm whether the operation was a success or a failure, an unreported exception will cause CloudFormation to wait until the operation times out before starting a stack rollback. If the exception occurs again on rollback, CloudFormation will wait again for a timeout before ultimately ending in a rollback failure. During this time, your stack is unusable, and timeout issues can be time-consuming to troubleshoot.

To avoid this, make sure that your function's code has logic to handle exceptions, the ability to log the failure to help you troubleshoot, and if needed, the ability to respond back to CloudFormation with an HTTPS response confirming that an operation failed."

In my lambda code , even when the code has a runtime exception , my lambda still returns a response to CloudFormation to avoid stack failure and prevent timeout errors which are very hard to debug.

Please see below an expert of a catch block in my code which shows that even in case of an error , a response is returned to CloudFormation.

catch (Exception ex)
{
context.Logger.LogLine($"StartupProgram::LoadMasterData => {ex.Message}");
context.Logger.LogLine($"StartupProgram::LoadMasterData => {ex.StackTrace}");

//Error - log it into the cloudformation console
CloudFormationResponse objResponse =
new CloudFormationResponse(
Constants.CloudformationErrorCode,
ex.Message,
context.LogStreamName,
request.StackId,
request.RequestId,
request.LogicalResourceId,
null
);

return objResponse.CompleteCloudFormationResponse(request, context).GetAwaiter().GetResult();
}

2.Set reasonable timeout periods, and report when they're about to be exceeded

"If an operation doesn't execute within its defined timeout period, the function raises an exception and no response is sent to CloudFormation.

To avoid this, ensure that the timeout value for your Lambda functions is set high enough to handle variations in processing time and network conditions. Consider also setting a timer in your function to respond to CloudFormation with an error when a function is about to timeout; this can help prevent function timeouts from causing custom resource timeouts and delays."

In this particular case , I have specified the lambda function timeout to 300 seconds (the maximum supported in lambda) to ensure that the lambda function does not timeout in any case. Because, if the function timeouts somehow , CloudFormation will not receive the response that it expects and the whole stack will get stuck . More on this later.

I have yet to figure out how to set a timer in a lambda function written in C# that can react when the timeout is about to expire , and at least return a sample response back to CloudFormation from the lambda , and in turn prevent the stack from getting stuck. Will update this post if I have any leads.

3.Understand and build around Create, Update, and Delete events

"Depending on the stack action, CloudFormation sends your function a Create, Update, or Delete event. Each event is handled distinctively, so you should ensure that there are no unintended behaviors when any of the three event types is received.

For more information, see Custom Resource Request Types.

For each of these different request types , CloudFormation injects different types of properties in the request object of the lambda event , and you need to handle them differently.

A sample class in C# that emulates the different properties that are present in the event object injected into the lambda function associated with the custom resource.

public class CloudFormationRequest
{
     public string StackId { get; set; }
     public string ResponseURL { get; set; }
     public string RequestType { get; set; }
     public string ResourceType { get; set; }
     public string RequestId { get; set; }
     public string LogicalResourceId { get; set; }
     public string PhysicalResourceId { get; set; } //valid for delete and update operations
     public object ResourceProperties { get; set; } //valid for delete and update operations
     public object OldResourceProperties { get; set; } //valid for update operations
}

4.Understand how CloudFormation identifies and replaces resources

"When an update triggers replacement of a physical resource, CloudFormation compares the PhysicalResourceId returned by your Lambda function to the previous PhysicalResourceId; if the IDs differ, CloudFormation assumes the resource has been replaced with a new physical resource.

However, the old resource is not implicitly removed to allow a rollback if necessary. When the stack update is completed successfully, a Delete event request is sent with the old ID as an identifier. If the stack update fails and a rollback occurs, the new physical ID is sent in the Delete event.

With this in mind, returning a new PhysicalResourceId should be done with care, and delete "events must consider the input PhysicalId to ensure that updates that require replacement are properly handled."

In my particular case , since I was not actually creating a custom resource of any kind , rather loading configuration data into a DynamoDB table , I did not have to consider the update/delete cases . I simply pushed the data into the table in case of Create request type.

if (string.Equals(request.RequestType, Constants.CloudFormationCreateRequestType))
{
dynamoDBHelper.putItemTable1(item1, request.ResourceProperties.TableName);

//Success - data inserted properly in the dynamoDB
CloudFormationResponse objResponse =
new CloudFormationResponse(
Constants.CloudformationSuccessCode,
"Custom Resource Creation Successful",
$"{request.StackId}-{request.LogicalResourceId}-DataLoad",
request.StackId,
request.RequestId,
request.LogicalResourceId,
item1
);

return objResponse.CompleteCloudFormationResponse(request, context).GetAwaiter().GetResult();
}
else
{
CloudFormationResponse objResponse =
new CloudFormationResponse(
Constants.CloudformationSuccessCode,
"Do nothing.Data will be pushed in only when stack event is Create",
context.LogStreamName
request.StackId,
request.RequestId,
request.LogicalResourceId,
null
);
return objResponse.CompleteCloudFormationResponse(request, context).GetAwaiter().GetResult();
}

5.Make sure that your functions are designed with idempotency in mind

"An idempotent function can be repeated any number of times with the same inputs, and the result will be the same as if it had been done only once. Idempotency is valuable when working with CloudFormation to ensure that retries, updates, and rollbacks don't cause the creation of duplicate resources, errors on rollback or delete, or other unintended effects.

For example, if CloudFormation invokes your function to create a resource, but doesn't receive a response that the resource was created successfully, CloudFormation might invoke the function again, resulting in the creation of a second resource; the first resource may become orphaned.

How to address this can differ depending on the action your function is intended to perform, but a common technique is to use a uniqueness token that CloudFormation can use to check for pre-existing resources. For example, a hash of the StackId and LogicalResourceId could be stored in the resource's metadata or in a DynamoDB table."

My code to make the functions idempotent - DynamoDB insert is only performed if the item does not exist before.

if (getItem(masterItem.UniqueID,TableName)== null) // this item does not exist
{
   Table table = Table.LoadTable(client, TableName);

   var clientItem = new Document();
   clientItem["UniqueID"] = masterItem.UniqueID;
   clientItem["EmployeeID"] = masterItem.EmployeeID;
   clientItem["Name"] = masterItem.Name;
   clientItem["Employee"] = masterItem.Designation;
   clientItem["Age"] = masterItem.Age;
   clientItem["Department"] = masterItem.Department;

   table.PutItemAsync(clientItem).GetAwaiter().GetResult();
   context.Logger.LogLine("DynamoDBHelper::PutItem() -- PutOperation succeeded");
}

6.Rollbacks

If a stack operation fails, CloudFormation attempts to roll back, reverting all resources to their prior state. This results in different behaviors depending on whether the update caused a resource replacement.

Ensuring that replacements are properly handled and the old resources are not implicitly removed until a delete event is received will help ensure that rollbacks are executed smoothly.

To help implement best practices when using custom resources, consider using the Custom Resource Helper provided by awslabs, which can assist with exception and timeout trapping, sending responses to CloudFormation, and logging.

Also , AWS Dotnet SDK does not support CloudFormation custom resources implicitly . So , you will need to create the custom classes yourself . I got a lot of help and ideas from this GitHub repo -> https://medium.com/@sch.bar/a-deep-dive-on-aws-cloudformation-custom-resources-72416f2e9cef

You can also check my code here at - https://github.com/turjachaudhuri/CF-custom-resources

Be sure to always return a response from lambda function , even if the code gets an error/ somehow timeout(s) . Otherwise the CloudFormation stack might get stuck and you will have to wait till the event timeout(s). You can find more discussion on that topic here-

https://forums.aws.amazon.com/thread.jspa?threadID=176003

Where to go from here?

This has endless opportunities . This is a great way to extend CloudFormation to create custom resources . I will extend my source project to achieve two things-

Setup a AWS RDS instance with sample SQL script for master data setup.
Load a DynamoDB table from a S3 file when the DB is created.

As always , please provide me feedback on how to improve , and let me know if there is anything else that you guys are working on in this regard.