Intro
This article is inspired by a blog post written by Yan Cui where he shows how we can cache AWS SSM Parameter Store values.
This approach is recommended so that we can re-use the same values during multiple invocations of our Lambda function and save costs of API calls to SSM Parameter Store.
The following is an example using AWS CDK where we define an SSM Parameter and use the aws-sdk to fetch the parameter and cache it for a given amount of time which we can configure.
Here's the repository for those who want to dive right in :)
ryands17 / lambda-ssm-cache
Cache SSM Paramter Store values in Lambda via CDK
Let's start by defining the parameter that we are going to fetch in our Lambda:
// lib/lambda-ssm-stack.ts
import * as ssm from '@aws-cdk/aws-ssm'
new ssm.StringParameter(this, 'dev-key1', {
parameterName: '/dev/key1',
stringValue: 'value1',
})
This is a snippet that creates a StringParameter
i.e. an un-encrypted value to store with the parameter path /dev/key1
.
In a project, it's recommended to store keys in a path based format like ${projectName}/${environment}/${keyName}
so that we can easily fetch all the values under the same path. For e.g. myProject/dev/DB_HOST
and myProject/prod/DB_HOST
for different environments.
Next, let's define our Lambda function:
// lib/lambda-ssm-stack.ts
import { join } from 'path'
import * as ln from '@aws-cdk/aws-lambda-nodejs'
const lambdaDir = join(__dirname, '..', 'lambda-fns')
const handler = new ln.NodejsFunction(this, 'fetchParams', {
runtime: lambda.Runtime.NODEJS_12_X,
memorySize: 512,
handler: 'handler',
entry: join(lambdaDir, 'src', 'index.ts'),
depsLockFilePath: join(lambdaDir, 'yarn.lock'),
nodeModules: ['ms'],
sourceMap: true,
})
In the above snippet, we have created a Lambda function using the Node.js 12 runtime allocating it a memory of 512 MB with some specific values. Let's have a look at these.
The
entry
option accepts a file in which we will define our code to fetch the above defined parameter. In this case, it's anindex.ts
file in a directory namedlambda-fns
which we will look at further.The
handler
option is the name of the function which is expored in ourindex.ts
file and that will be used by Lambda.The
depsLockFilePath
,nodeModules
, andsourceMap
options are for bundling our Node.js function before deploying. This is done using the aws-lambda-nodejs package that transpiles our TypeScript code to JavaScript withnode_modules
so that it can run it on Lambda.
We're done creating the function. Let's look at the final snippet of code required to deploy our stack:
// lib/lambda-ssm-stack.ts
handler.addToRolePolicy(
new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
actions: ['ssm:GetParametersByPath'],
resources: [`arn:aws:ssm:${this.region}:*:parameter/dev*`],
})
)
This final snippet allows the Lambda function access to fetch the created parameter from the Parameter Store, and not just any parameter but the one starting with /dev
. This makes sure that we do not fetch any unwanted/unneeded secret values especially for other projects.
Now let's look at our Lambda function in the lambda-fns
folder:
// lambda-fns/index.ts
import { Context } from 'aws-lambda'
import { config, loadParameters } from './config'
export const handler = async (event: any, context: Context) => {
await loadParameters()
return {
value: config.values['/dev/key1'],
success: true,
}
}
This is our handler
function that calls the loadParameters
function to fetch our parameter and returns the value.
Let's have a look at config.ts
where the main code for fetching the parameter and caching is included.
We'll start by defining the variables required:
// lambda-fns/config.ts
import { SSM } from 'aws-sdk'
import ms from 'ms'
type Config = {
values: Record<string, string | undefined>
expiryDate?: Date
}
const ssm = new SSM()
export let config: Config = { values: {} }
Here we have initialized an instance of SSM
which we will use to fetch the parameter and the config
variable that will be used to store our parameter. We can have multiple values fetched as well that can be stored here.
Notice the type of the config
variable. It has two keys, the values
which will store our parameters and the expiryDate
that will invalidate our cache and fetch all the parameters again.
The expiry time is configurable and we will see how this is used in the loadParameters
function next.
// lambda-fns/config.ts
export const loadParameters = async ({
expiryTime: cacheDuration = '1h',
}: {
expiryTime?: string
} = {}) => {
if (!config.expiryDate) {
config.expiryDate = new Date(Date.now() + ms(cacheDuration))
}
if (isConfigNotEmpty() && !hasCacheExpired()) return
console.log('[Cost]: API called')
config.values = {}
const { Parameters = [] } = await ssm
.getParametersByPath({
Path: '/dev',
})
.promise()
for (let param of Parameters) {
if (param.Name) config.values[param.Name] = param.Value
}
}
const hasCacheExpired = () =>
config.expiryDate && new Date() > config.expiryDate
const isConfigNotEmpty = () => Object.keys(config.values).length
The final and most important function of our Lambda, the loadParameters
function accepts a single value named expiryTime
that is by default set to 1 hour and can be overridden. I have used the ms library to set human readable time periods which will be automatically converted to milliseconds.
We first check if the expiry date exists. If not, we set it to the value: current time + expiryTime.
Then we have this snippet:
if (isConfigNotEmpty() && !hasCacheExpired()) return
The following indicates that if we have values present in our config and if the cache hasn't expired then bail out of the function as we already have the values and we wouldn't want to call the API to fetch our parameters.
The above functions were just added to make the condition readable and they are defined as follows:
const hasCacheExpired = () =>
config.expiryDate && new Date() > config.expiryDate
const isConfigNotEmpty = () => Object.keys(config.values).length
The next snippet is the entire fetching and setting of the parameters as we have already checked for the above conditions and we know that either the cache has expired or we do not have any values in our config.
console.log('[Cost]: API called')
config.values = {}
const { Parameters = [] } = await ssm
.getParametersByPath({
Path: '/dev',
})
.promise()
for (let param of Parameters) {
if (param.Name) config.values[param.Name] = param.Value
}
In this, we simply added a log to know whether we're fetching the parameters and we have set the values
to an empty object ({}
). Then, we call the getParametersByPath
method and pass the path dev
we created in our resources file. Lastly, we loop over the parameters and add those to the values
property in our config.
And we're done! To deploy this stack, run yarn cdk deploy
or npm run cdk deploy
.
Note: A prerequisite for this would be installing aws-cli and configuring the profile (I have configured the default
profile in this case) with the Access and Secret keys.
Now, let's run this lambda by creating a test event from the console:
On clicking Test, we can see the logs and on the first run, we would see something like this:
We successfully get the result of the parameter we stored and if we check the logs, we can see API CALLED because the value wasn't present as it's the very first invocation and the call to Parameter Store was made to fetch the value.
Let's run the test event again and see the result.
We get the value, but notice that the log API CALLED isn't present, which means that the value was obtained from the cache :)
This values remains due the Lambda reusing the execution environment on subsequent invocations. This is although upto Lambda and we cannot configure this externally. Lambda can start on a clean state and we would need to add a condition to fetch the values again which we have done in the loadParameters
function as seen above.
So that was it for caching values from the SSM Parameter Store in a Lambda function. Do not forget to delete this stack after you've done with it using yarn cdk destroy
.
Thank you all for reading! Do like and share this post if you've enjoyed it :)
Top comments (3)
First of all, great article!
Do you need the caching logic you've described here? I would think you should simply request the parameters outside of the lambda event handler so that for the life of the lambda you have access to the parameters. Once you don't have enough requests, that lambda will be recycled and upon another request the lambda will be spun up again.
I guess the only time you'd need the caching logic is if you expect the lambda to live longer than an hour which is possible.
Thanks a lot!
The caching logic can be kept smaller as well so even if the Lambda execution environment retained, you can refresh those secrets in cases you need them to update in a specific time period.