Using CloudFormation Custom Resources to react to coded config changes with AWS Lambda

As ever, engineering in a startup is a matter of working out when you can fully invest in a feature, and when to bridge the gap until a more optimal time arises as the scale doesn't make sense yet and there are more important problems to focus on.

One particular set of examples of this we've faced here at Pelago (formerly Quit Genius) is when can you take expanding config out of code and into a simple CRUD system, and when do you have to keep it there. Adding this CRUD system into production typically requires both backend and frontend resources, and having the spare time in both teams is hard to come by!

Therefore, when given the scenario of “we need to tell a third party when a new element is added to a given config enum”, we had to go to the drawing board to prevent further steps being added to the SOP of expanding this given piece of config. I'd seen examples of using CloudFormation Custom Resources for doing on-stack-changes to set up or update other resources - e.g. filling default entries in a DynamoDB tables - and thought that it could be a solution here too.

Having then had to cobble together different pieces of advice in order to get the end solution to work, I thought I’d write this guide so the next person who faces this problem can tackle it even quicker!

How to set up serverless config file.

First off the easy bit. I’m most familiar with using serverless.ts files these days, but the same config should be easily translated to serverless.yml - just the function capability of the custom InputParameter gets a bit more tricky!

The Lambda (which I’ll get into below) can be declared just like a normal serverless function with no trigger.

functions: {
  ...
  myConfigUpdater: {
    handler: 'handler.myConfigUpdater',
    timeout: 90
  }
}

The fun bit is that you then need to declare a custom resource that will point at that Lambda as the user of its ServiceToken. This is done by adding a resource of type AWS::CloudFormation::CustomResource. As noted in the docs - this can actually be modified to a different Custom name if required.

The properties that aren’t the ServiceToken are entirely customisable here. For simplicity I’ve called mine InputParameter and its presence here means that every time the value calculated there doesn’t match up with the most recent value that CloudFormation has (from the last time you deployed the service), that CloudFormation will re-run the Custom Resource, because it sees the property(s) of the resource have changed. Here, the getSumOfMyConfigEnum is a JavaScript function in a shared library that every time the config enum I’m using is added to the sum of the enum values changes. This change therefore means the Custom Resource is re-run when the config enum changes. If you were to want your Lambda to run every time the service is deployed you could chose to do something like InputParameter: Date.now() to just get the most recent Epoch time value. (Presuming you’re deploying less frequently than once a second! 😀)

Worth noting here that the name of the Resource (MyConfigUpdaterResource) doesn’t actually matter config-wise - it’s just for you to keep track of your resources. Also, as commented in the code, you need to change the name of the Lambda function in both the Fn::GetAtt and DependsOn statements to allow for the CloudFormation to attach.

resources: {
      Resources: {
        // this is a custom resource that will update the config when triggered by a change in the input parameter
        // when this parameter changes it tells CloudFormation that a change has occurred and thus the lambda needs to be triggered
        // see further details in the README.
        MyConfigUpdaterResource: {
          Type: 'AWS::CloudFormation::CustomResource',
          Properties: {
            ServiceToken: {
              'Fn::GetAtt': ['MyConfigUpdaterLambdaFunction', 'Arn'], // N.B. if the function name above is changed this needs to be matched.
            },
            InputParameter: getSumOfMyConfigEnum(),
          },
          DependsOn: ['MyConfigUpdaterLambdaFunction'], // N.B. if the function name is changed above this needs to be matched.
        },
      },
    },

Problem: CloudFormation Hanging? Can’t get a response to work successfully?

Now comes the real fun part, getting the code to work alongside the serverless config. Given if you get it completely wrong you can easily be sat with the CloudFormation stack hanging for 60 minutes to timeout, it can be quite frustrating!

Response format

The first issue is the response format. There is a handly little module that’s been built by AWS for handling the response, but as documented in the CloudFormation docs it’s only included when you use the ZipFile property to write your code!

⚠️ “For code in buckets, you must write your own functions to send responses.”

Therefore you need to either copy across the code directly from AWS - or rely on a package such as this one. Though as documented on GitHub, if AWS changes their code then that package needs to be informed of the change - not exactly ideal!

Async Changes

Having made sure I was using the cfn-response package I thought I had finally gotten past the difficult part of the challenge. Little did I know that in an even more obscure part of the internet is a Stack Overflow thread pointing out that this in fact won’t work if you have any async operations happening inside your Custom Resource. This is because the package from AWS is designed to work synchronously. Given I needed to make an API call to our third party as a part of my config changes, this wasn’t going to work for me!

I then found this package called cfn-response-async which solved the issue. The below code is thus a simplified version of what I’ve settled upon. If my async config changes are made successfully in our third party, then we return a SUCCESS and the CloudFormation succeeds. If there’s an error, this is caught and the CloudFormation deployment is stopped.

Further logic can be inserted here to change what’s being done as the handler will be invoked with three different types of RequestType on the event object - these are Create, Update and Delete. Thus you can use a switch statement if you need to do something more complex (e.g. a setup on the create, and then a deconstruction of something on the delete)

Final Pseudocode

import * as cloudFormationResponse from 'cfn-response-async';
...

const customResourcehandler = async (event, context) => {
  try {
    log.info(`Starting custom resource handler`);
    // make the config changes
    await makeMyAsyncConfigChanges()
    await cloudFormationResponse.send(
      event,
      context,
      'SUCCESS',
      {
        Input: event.ResourceProperties.InputParameter,
      },
      event.StackId
    );
    // respond with a positive 
  } catch (error) {
    log.error(`Error updating config: ${error.message}`)
    await cloudFormationResponse.send(
      event, 
      context,
      'FAILED',
      {
        Input: event.ResourceProperties.InputParameter,
      },
      event.StackId
    )
  }
}

Conclusion

So there you have it - a couple of wrinkles to get asynchronous changes done by CloudFormation, but super useful to help us save time in development!