Separate NPM Dependencies into layers
One way to improve deployment time when developing AWS Lambda functions is to separate your function's dependencies into layers. You can save time by deploying the layers once so that the subsequent deployments of your main functions code are as lightweight as possible.
Serverless Framework has built-in support for deploying layers by creating your desired dependencies in a separated package.json file and putting the file in the nodejs directory under the layer path.
# directory structure
layers/
stablelibs/
nodejs/
package.json
otherLibs/
nodejs/
package.json
serverless.yml
package.json
# serverless.yml
layers:
commonlibs:
path: layers/commonlibs
functions:
...
The problems
However, this method has some problems.
You have to maintain multiple
package.jsonfiles. It will be very difficult to keep the dependencies version in sync in every file, especially if you are using Github'sdependabotto keep updating the dependencies version.It will involve multiple manual steps for each deployment. For example, you have to go into each layer and run
yarn installto populatenode_modulesbefore running the actual deploy command.The deployment time can be even worst because it now has to take time to install dependencies and package each layer for every deployment.
For the second and third points, the existing plugin such as serverless-layer is a good solution, but in exchange, you will lose some built-in features such as the ability to exclude unnecessary files inside node_modules for some packages.
Different projects have different requirements for the workflow. It would be unfortunate to let the framework or plugins limit what you want to do.
I have been researching this problem, and It turns out that Serverless Framework is more flexible and powerful than I thought. It is very easy to customise the deployment workflow anyhow I like. And what follows is an example of what I have done with my current project.
1. Separate layers deployment from functions
In my project, I already have multiple serverless.yml for each of my API gateways. For example, I have serverless.internal-api.yml and serverless.external-api.yml for my internal and external API gateways.
The functions in both files share the same dependencies. So I create another file dedicated to the layers only.
serverless.internal-api.yml
serverless.external-api.yml
serverless.layers.yml <-- this
package.json
Let's start with the simple version of serverless.layers.yml
service: dependency-layers
layers:
awslibs:
path: layers/awslibs
name: ${self:service}-${sls:stage}-awslibs
platformlibs:
path: layers/platformlibs
name: ${self:service}-${sls:stage}-platformlibs
stablelibs:
path: layers/stablelibs
name: ${self:service}-${sls:stage}-stablelibs
otherlibs:
path: layers/otherlibs
name: ${self:service}-${sls:stage}-otherlibs
And then use these layers with the functions in serverless.internal-api.yml and serverless.external-api.yml
package:
individually: true
# include only the functions code in the dist folder
patterns:
- "!**"
- "dist/**"
providers:
layers:
- arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-awslibs:latest
- arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-platformlibs:latest
- arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-stablelibs:latest
- arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-otherlibs:latest
plugins:
# This plugin will replace the 'latest' after the ARNs to the actual version number
- serverless-latest-layer-version
Now I can deploy the layers independently from the functions. I have this npm script in package.json
scripts: {
"deploy:layer": "serverless deploy -c serverless.layers.yml",
"deploy:internal-api": "serverless deploy -c serverless.internal-api.yml",
"deploy:external-api": "serverless deploy -c serverless.external-api.yml"
}
2. Use hooks to populate layer's files at built-time
Notice that we will need the following folder structure for the layer files.
# directory structure
layers/
awslibs/
nodejs/
node_modules
package.json
platformlibs/
nodejs/
node_modules
package.json
stablelibs/
nodejs/
node_modules
package.json
otherlibs/
nodejs/
node_modules
package.json
serverless.internal-api.yml
serverless.external-api.yml
serverless.layers.yml
package.json
We will populate these files dynamically when running the deploy:layers script. To do that, I insert dependencyIncludeRegexPatterns for each layer configuration to tell which dependencies to be copied from the main package.json to the layer's package.json.
service: dependency-layers
layers:
awslibs:
path: layers/awslibs
name: ${self:service}-${sls:stage}-awslibs
installCommand: yarn install
dependencyIncludeRegexPatterns:
- "^@aws-sdk"
- "^aws"
platformlibs:
path: layers/platformlibs
name: ${self:service}-${sls:stage}-platformlibs
installCommand: yarn install
dependencyIncludeRegexPatterns:
- "^@opentelemetry"
- "^@nestjs"
stablelibs:
path: layers/stablelibs
name: ${self:service}-${sls:stage}-stablelibs
dependencyIncludeRegexPatterns:
- "^date-fns$"
- "^rxjs$"
otherlibs:
path: layers/otherlibs
name: ${self:service}-${sls:stage}-otherlibs
dependencyIncludeRegexPatterns:
- "."
Then I use serverless-plugin-scripts to define a hook script that will run right after we kick off the script deploy:layer.
plugins:
- serverless-plugin-scripts
custom:
scripts:
commands:
hooks:
"before:package:initialize": ts-node tools/initializeLayerDependencies.ts serverless.layers.yml
Basically, we can do anything in tools/initializeLayerDependencies.ts, but here I populate the package.json for each layer and run the installCommand
import { mkdirSync, readFileSync, rmSync, writeFileSync } from "fs";
import * as yaml from "js-yaml";
import packageJson from "../package.json";
import { run } from "./libs/process-utils";
const configFile = process.argv[2];
initializeLayerDependencies(configFile);
// ################################################################
interface LayerConfig {
path: string;
installCommand: string;
yarnCacheFolder: string;
dependencyIncludeRegexPatterns: string[];
dependencyExcludeRegexPatterns?: string[];
}
async function initializeLayerDependencies(configFile: string) {
const config = yaml.load(readFileSync(configFile, "utf8")) as {
layers: { [layerName: string]: LayerConfig };
};
const remainingDependencies = packageJson.dependencies;
for (const [_layerName, layerConfig] of Object.entries(config.layers)) {
console.log({ _layerName, layerConfig });
if (!validConfig(layerConfig)) {
console.warn("Invalid layerConfig!");
continue;
}
const dependencyPath = `${layerConfig.path}/nodejs`;
rmSync(dependencyPath, { recursive: true, force: true });
console.log(`Initializing dependencies for ${layerConfig.path} ...`);
const layerDependencies = Object.keys(remainingDependencies).reduce(
(filteredDeps, key) => {
if (
shouldInclude(
key,
layerConfig.dependencyIncludeRegexPatterns,
layerConfig.dependencyExcludeRegexPatterns,
)
) {
filteredDeps[key] = remainingDependencies[key];
delete remainingDependencies[key];
}
return filteredDeps;
},
{},
);
mkdirSync(dependencyPath, { recursive: true });
writeFileSync(
`${dependencyPath}/package.json`,
JSON.stringify({ dependencies: layerDependencies }, null, 4),
);
writeFileSync(`${dependencyPath}/yarn.lock`, "");
await run(layerConfig.installCommand, dependencyPath, {
YARN_CACHE_FOLDER: layerConfig.yarnCacheFolder,
});
}
}
function shouldInclude(
packageName: string,
includePatterns?: string[],
excludePatterns?: string[],
) {
if (
excludePatterns &&
excludePatterns.some((regex) => packageName.match(regex))
) {
return false;
}
if (
includePatterns &&
includePatterns.some((regex) => packageName.match(regex))
) {
return true;
}
return false;
}
function validConfig({
path,
yarnCacheFolder,
installCommand,
dependencyIncludeRegexPatterns,
}: LayerConfig) {
return (
path &&
yarnCacheFolder &&
installCommand &&
dependencyIncludeRegexPatterns &&
true
);
}
export async function run(
cmd: string,
cwd: string,
env: Record<string, string> = {},
) {
try {
execSync(cmd, {
cwd,
env: {
...process.env,
...env,
},
maxBuffer: 1024 * 1024 * 500,
stdio: "inherit",
});
} catch (error) {
console.log(error.stdout?.toString());
throw error;
}
}
3. Improve installment time with yarn cache
We can further speed up the yarn install by utilizing yarnCacheFolder
service: dependency-layers
layers:
awslibs:
path: layers/awslibs
name: ${self:service}-${sls:stage}-awslibs
installCommand: yarn install
yarnCacheFolder: ../../../.yarn/cache <-- here
dependencyIncludeRegexPatterns:
- "^@aws-sdk"
- "^aws"
4. Save the dependencies and skip deployment
Imagine the sequence to run in our CI/CD would be like
- yarn install
- yarn deploy:layer # serverless deploy -c serverless.layers.yml
- yarn deploy:internal-api # serverless deploy -c serverless.internal-api.yml
- yarn deploy:external-api # serverless deploy -c serverless.external-api.yml
So far so good, but it still takes non-trivial time to build the layers. Serverless Framework can check the deployment hash and skip the deployment, but it does not know when to skip the build (packaging) step.
This is easy. First, we will cache the dependencies from the main package.json in the SSM parameter store together with the deployment package.
# serverless.layers.yml
resources:
Resources:
LayerDependencies:
Type: AWS::SSM::Parameter
Properties:
Type: String
Name: /${self:service}/${sls:stage}/dependencies
Value: ${file(./tools/getDependenciesAsString.js):value}
# tools/getDependenciesAsString.js
module.exports = async () => {
const fs = require("fs");
const packageJson = JSON.parse(fs.readFileSync("package.json", "utf-8"));
return {
value: JSON.stringify(packageJson.dependencies),
};
};
Next, we can create a command to compare the cached dependencies with the current dependencies in the main package.json and run yarn deploy:layer when we detect the diff.
# serverless.layers.yml
custom:
scripts:
commands:
update: ts-node tools/updateDependencyLayers.ts "/${self:service}/${sls:stage}/dependencies" ap-southeast-1 "yarn deploy:lambda:layer"
hooks:
"before:package:initialize": ts-node tools/initializeLayerDependencies.ts serverless.layers.yml
# tools/updateDependencyLayers.ts
import packageJson from "../package.json";
import _ from "lodash";
import { GetParameterCommand, SSMClient } from "@aws-sdk/client-ssm";
import { run } from "./libs/process-utils";
main(process.argv[2], process.argv[3], process.argv[4]);
async function main(ssmPath: string, region: string, deployCommand: string) {
const ssm = new SSMClient({ region });
const response = await ssm.send(
new GetParameterCommand({
Name: ssmPath,
}),
);
const cachedDependenciesString = response.Parameter.Value;
if (!cachedDependenciesString) {
console.log("Not found dependencies in cache, should deploy layers");
await run(deployCommand, ".");
return;
}
const cachedDependencies = JSON.parse(cachedDependenciesString);
if (!_.isEqual(cachedDependencies, packageJson.dependencies)) {
console.log(
"Found dependencies in cache and is not equal to dependencies in package.json, should deploy layers",
);
await run(deployCommand, ".");
return;
}
console.log("The dependencies are up-to-date, should skip deploy layers");
}
Now we can run npx serverless update -c serverless.layers.yml to invoke this script.
Conclusion
It will be much faster for developers to edit the function code and deploy it because the function code is lightweight.
Only when adding new libraries or updating their versions should we then deploy the layers.
The CI/CD pipeline can always run the update, which will skip the deployment if the dependencies are up-to-date.
Top comments (0)