DEV Community

Pipat Sampaokit
Pipat Sampaokit

Posted on

Customize the deployment of AWS Lambda Layer anyhow you like

#aws #serverless #devops

Separate NPM Dependencies into layers

One way to improve deployment time when developing AWS Lambda functions is to separate your function's dependencies into layers. You can save time by deploying the layers once so that the subsequent deployments of your main functions code are as lightweight as possible.

Serverless Framework has built-in support for deploying layers by creating your desired dependencies in a separated package.json file and putting the file in the nodejs directory under the layer path.

# directory structure
layers/
  stablelibs/
    nodejs/
      package.json
  otherLibs/
    nodejs/
      package.json  
serverless.yml
package.json
Enter fullscreen mode Exit fullscreen mode 
# serverless.yml

layers:
  commonlibs:
    path: layers/commonlibs

functions:
  ...
Enter fullscreen mode Exit fullscreen mode

The problems

However, this method has some problems.

  1. You have to maintain multiple package.json files. It will be very difficult to keep the dependencies version in sync in every file, especially if you are using Github's dependabot to keep updating the dependencies version.

  2. It will involve multiple manual steps for each deployment. For example, you have to go into each layer and run yarn install to populate node_modules before running the actual deploy command.

  3. The deployment time can be even worst because it now has to take time to install dependencies and package each layer for every deployment.

For the second and third points, the existing plugin such as serverless-layer is a good solution, but in exchange, you will lose some built-in features such as the ability to exclude unnecessary files inside node_modules for some packages.

Different projects have different requirements for the workflow. It would be unfortunate to let the framework or plugins limit what you want to do.

I have been researching this problem, and It turns out that Serverless Framework is more flexible and powerful than I thought. It is very easy to customise the deployment workflow anyhow I like. And what follows is an example of what I have done with my current project.

1. Separate layers deployment from functions

In my project, I already have multiple serverless.yml for each of my API gateways. For example, I have serverless.internal-api.yml and serverless.external-api.yml for my internal and external API gateways.

The functions in both files share the same dependencies. So I create another file dedicated to the layers only.

serverless.internal-api.yml
serverless.external-api.yml
serverless.layers.yml <-- this
package.json
Enter fullscreen mode Exit fullscreen mode

Let's start with the simple version of serverless.layers.yml

service: dependency-layers
layers:
  awslibs:
    path: layers/awslibs
    name: ${self:service}-${sls:stage}-awslibs
  platformlibs:
    path: layers/platformlibs
    name: ${self:service}-${sls:stage}-platformlibs
  stablelibs:
    path: layers/stablelibs
    name: ${self:service}-${sls:stage}-stablelibs
  otherlibs:
    path: layers/otherlibs
    name: ${self:service}-${sls:stage}-otherlibs
Enter fullscreen mode Exit fullscreen mode

And then use these layers with the functions in serverless.internal-api.yml and serverless.external-api.yml


package:
  individually: true

  # include only the functions code in the dist folder
  patterns:
    - "!**"
    - "dist/**"

providers:
  layers:
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-awslibs:latest
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-platformlibs:latest
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-stablelibs:latest
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-otherlibs:latest

plugins:
  # This plugin will replace the 'latest' after the ARNs to the actual version number
  - serverless-latest-layer-version
Enter fullscreen mode Exit fullscreen mode

Now I can deploy the layers independently from the functions. I have this npm script in package.json

scripts: {
  "deploy:layer": "serverless deploy -c serverless.layers.yml",
  "deploy:internal-api": "serverless deploy -c serverless.internal-api.yml",
  "deploy:external-api": "serverless deploy -c serverless.external-api.yml"
}
Enter fullscreen mode Exit fullscreen mode

2. Use hooks to populate layer's files at built-time

Notice that we will need the following folder structure for the layer files.

# directory structure
layers/
  awslibs/
    nodejs/
      node_modules
      package.json
  platformlibs/
    nodejs/
      node_modules
      package.json  
  stablelibs/
    nodejs/
      node_modules
      package.json  
  otherlibs/
    nodejs/
      node_modules
      package.json  
serverless.internal-api.yml
serverless.external-api.yml
serverless.layers.yml
package.json
Enter fullscreen mode Exit fullscreen mode

We will populate these files dynamically when running the deploy:layers script. To do that, I insert dependencyIncludeRegexPatterns for each layer configuration to tell which dependencies to be copied from the main package.json to the layer's package.json.

service: dependency-layers
layers:
  awslibs:
    path: layers/awslibs
    name: ${self:service}-${sls:stage}-awslibs
    installCommand: yarn install
    dependencyIncludeRegexPatterns:
      - "^@aws-sdk"
      - "^aws"
  platformlibs:
    path: layers/platformlibs
    name: ${self:service}-${sls:stage}-platformlibs
    installCommand: yarn install
    dependencyIncludeRegexPatterns:
      - "^@opentelemetry"
      - "^@nestjs"
  stablelibs:
    path: layers/stablelibs
    name: ${self:service}-${sls:stage}-stablelibs

    dependencyIncludeRegexPatterns:
      - "^date-fns$"
      - "^rxjs$"
  otherlibs:
    path: layers/otherlibs
    name: ${self:service}-${sls:stage}-otherlibs
    dependencyIncludeRegexPatterns:
      - "."
Enter fullscreen mode Exit fullscreen mode

Then I use serverless-plugin-scripts to define a hook script that will run right after we kick off the script deploy:layer.

plugins:
  - serverless-plugin-scripts

custom:
  scripts:
    commands:
      hooks:
        "before:package:initialize": ts-node tools/initializeLayerDependencies.ts serverless.layers.yml
Enter fullscreen mode Exit fullscreen mode

Basically, we can do anything in tools/initializeLayerDependencies.ts, but here I populate the package.json for each layer and run the installCommand

import { mkdirSync, readFileSync, rmSync, writeFileSync } from "fs";
import * as yaml from "js-yaml";
import packageJson from "../package.json";
import { run } from "./libs/process-utils";

const configFile = process.argv[2];

initializeLayerDependencies(configFile);

// ################################################################

interface LayerConfig {
    path: string;
    installCommand: string;
    yarnCacheFolder: string;
    dependencyIncludeRegexPatterns: string[];
    dependencyExcludeRegexPatterns?: string[];
}

async function initializeLayerDependencies(configFile: string) {
    const config = yaml.load(readFileSync(configFile, "utf8")) as {
        layers: { [layerName: string]: LayerConfig };
    };

    const remainingDependencies = packageJson.dependencies;
    for (const [_layerName, layerConfig] of Object.entries(config.layers)) {
        console.log({ _layerName, layerConfig });

        if (!validConfig(layerConfig)) {
            console.warn("Invalid layerConfig!");
            continue;
        }
        const dependencyPath = `${layerConfig.path}/nodejs`;
        rmSync(dependencyPath, { recursive: true, force: true });
        console.log(`Initializing dependencies for ${layerConfig.path} ...`);
        const layerDependencies = Object.keys(remainingDependencies).reduce(
            (filteredDeps, key) => {
                if (
                    shouldInclude(
                        key,
                        layerConfig.dependencyIncludeRegexPatterns,
                        layerConfig.dependencyExcludeRegexPatterns,
                    )
                ) {
                    filteredDeps[key] = remainingDependencies[key];
                    delete remainingDependencies[key];
                }

                return filteredDeps;
            },
            {},
        );

        mkdirSync(dependencyPath, { recursive: true });
        writeFileSync(
            `${dependencyPath}/package.json`,
            JSON.stringify({ dependencies: layerDependencies }, null, 4),
        );

        writeFileSync(`${dependencyPath}/yarn.lock`, "");
        await run(layerConfig.installCommand, dependencyPath, {
            YARN_CACHE_FOLDER: layerConfig.yarnCacheFolder,
        });
    }
}

function shouldInclude(
    packageName: string,
    includePatterns?: string[],
    excludePatterns?: string[],
) {
    if (
        excludePatterns &&
        excludePatterns.some((regex) => packageName.match(regex))
    ) {
        return false;
    }

    if (
        includePatterns &&
        includePatterns.some((regex) => packageName.match(regex))
    ) {
        return true;
    }

    return false;
}

function validConfig({
    path,
    yarnCacheFolder,
    installCommand,
    dependencyIncludeRegexPatterns,
}: LayerConfig) {
    return (
        path &&
        yarnCacheFolder &&
        installCommand &&
        dependencyIncludeRegexPatterns &&
        true
    );
}

export async function run(
    cmd: string,
    cwd: string,
    env: Record<string, string> = {},
) {
    try {
        execSync(cmd, {
            cwd,
            env: {
                ...process.env,
                ...env,
            },
            maxBuffer: 1024 * 1024 * 500,
            stdio: "inherit",
        });
    } catch (error) {
        console.log(error.stdout?.toString());
        throw error;
    }
}
Enter fullscreen mode Exit fullscreen mode

3. Improve installment time with yarn cache

We can further speed up the yarn install by utilizing yarnCacheFolder

service: dependency-layers
layers:
  awslibs:
    path: layers/awslibs
    name: ${self:service}-${sls:stage}-awslibs
    installCommand: yarn install
    yarnCacheFolder: ../../../.yarn/cache <-- here
    dependencyIncludeRegexPatterns:
      - "^@aws-sdk"
      - "^aws"
Enter fullscreen mode Exit fullscreen mode

4. Save the dependencies and skip deployment

Imagine the sequence to run in our CI/CD would be like

  - yarn install
  - yarn deploy:layer # serverless deploy -c serverless.layers.yml
  - yarn deploy:internal-api # serverless deploy -c serverless.internal-api.yml
  - yarn deploy:external-api # serverless deploy -c serverless.external-api.yml
Enter fullscreen mode Exit fullscreen mode

So far so good, but it still takes non-trivial time to build the layers. Serverless Framework can check the deployment hash and skip the deployment, but it does not know when to skip the build (packaging) step.

This is easy. First, we will cache the dependencies from the main package.json in the SSM parameter store together with the deployment package.

# serverless.layers.yml

resources:
    Resources:
        LayerDependencies:
            Type: AWS::SSM::Parameter
            Properties:
                Type: String
                Name: /${self:service}/${sls:stage}/dependencies
                Value: ${file(./tools/getDependenciesAsString.js):value}
Enter fullscreen mode Exit fullscreen mode 
# tools/getDependenciesAsString.js

module.exports = async () => {
    const fs = require("fs");
    const packageJson = JSON.parse(fs.readFileSync("package.json", "utf-8"));
    return {
        value: JSON.stringify(packageJson.dependencies),
    };
};
Enter fullscreen mode Exit fullscreen mode

Next, we can create a command to compare the cached dependencies with the current dependencies in the main package.json and run yarn deploy:layer when we detect the diff.

# serverless.layers.yml

custom:
    scripts:
        commands:
            update: ts-node tools/updateDependencyLayers.ts "/${self:service}/${sls:stage}/dependencies" ap-southeast-1 "yarn deploy:lambda:layer"
        hooks:
            "before:package:initialize": ts-node tools/initializeLayerDependencies.ts serverless.layers.yml
Enter fullscreen mode Exit fullscreen mode 
# tools/updateDependencyLayers.ts

import packageJson from "../package.json";
import _ from "lodash";
import { GetParameterCommand, SSMClient } from "@aws-sdk/client-ssm";
import { run } from "./libs/process-utils";

main(process.argv[2], process.argv[3], process.argv[4]);

async function main(ssmPath: string, region: string, deployCommand: string) {
    const ssm = new SSMClient({ region });
    const response = await ssm.send(
        new GetParameterCommand({
            Name: ssmPath,
        }),
    );
    const cachedDependenciesString = response.Parameter.Value;

    if (!cachedDependenciesString) {
        console.log("Not found dependencies in cache, should deploy layers");
        await run(deployCommand, ".");
        return;
    }
    const cachedDependencies = JSON.parse(cachedDependenciesString);
    if (!_.isEqual(cachedDependencies, packageJson.dependencies)) {
        console.log(
            "Found dependencies in cache and is not equal to dependencies in package.json, should deploy layers",
        );
        await run(deployCommand, ".");
        return;
    }

    console.log("The dependencies are up-to-date, should skip deploy layers");
}
Enter fullscreen mode Exit fullscreen mode

Now we can run npx serverless update -c serverless.layers.yml to invoke this script.

Conclusion

It will be much faster for developers to edit the function code and deploy it because the function code is lightweight.

Only when adding new libraries or updating their versions should we then deploy the layers.

The CI/CD pipeline can always run the update, which will skip the deployment if the dependencies are up-to-date.

Top comments (0)

Read next

garciadiazjaime profile image

React + AWS Cognito: Email Authentication Setup Guide (First Part)

Jaime -

frederickollinger profile image

Using a Self-Signed Certificate with Git Clone Https

Frederick Ollinger -

deekshagunde profile image

Cross-Project Dependencies Handling with DBT in AWS MWAA

Deeksha Gunde -

blessingglobally profile image

LINUX CHALLENGE DAY 2

Blessing Ndubuisi -