DEV Community

Managing Multiple Functions with AWS SAM and Webpack

Matt Morgan on February 10, 2020

I've been spending some quality time with the Serverless Application Model from AWS. I'm really enjoying this tool and I see it as a fantastic way ...

Read full post

Sandino • Jun 16 '20 • Edited

Thanks for sharing this is a great post!!.

Regarding your concern about your approach "adding a new function means touching both your template.yaml and your webpack.config.ts files." someone else figured out how to keep them in sync:

// Extract the AWS::Serverless::Function Resources as they
// are the entires we need to compile.
const { Resources } = CloudFormation.yamlParse(fs.readFileSync('template.yml'))

const entries = Object.values(Resources)
  .filter(resource => resource.Type == 'AWS::Serverless::Function')
  .filter(resource => resource.Properties.Runtime.startsWith('nodejs'))

  .map(resource => {
    const file = resource.Properties.Handler.split('.')[0]
    const prefix = resource.Properties.CodeUri.substr(9)

    return {
      name: `${prefix}/${file}`,
      entry: `${prefix}/${file}.ts`,
    }
  })

  .reduce((accumulator, resource) => {
    const { name, entry } = resource

    return Object.assign(accumulator, {
      [name]: path.resolve(entry),
    })
  }, {})

module.exports = {
  entry: entries,
  ...
}

Here is the full config: gist.github.com/henrikbjorn/d2eef5...

I think combining both approaches is a great toolchain to have.

Thank you.

Nick Bordeau • Jul 23 '20

Here's the Typescript version as well.

/** Interface for AWS SAM Function */
interface ISamFunction {
  Type: string;
  Properties: {
    AssumeRolePolicyDocument?: JSON;
    AutoPublishAlias?: string;
    AutoPublishCodeSha256?: string;
    CodeUri?: string;
    Description?: string;
    Environment?: {
      Variables: {
        [key: string]: string;
      };
    };
    Events?: EventSource;
    FunctionName?: string;
    Handler: string;
    Layers?: { [Ref: string]: string }[];
    Runtime: string;
    Timeout?: number;
    Tracing?: string;
    VersionDescription?: string;
  };
}

const { resources } = yamlParse(readFileSync(conf.templatePath, 'utf-8'));

const entries = Object.values(resources)

  .filter((resource: ISamFunction) => resource.Type === 'AWS::Serverless::Function')

  .filter(
    (resource: ISamFunction) =>
      resource.Properties.Runtime && resource.Properties.Runtime.startsWith('nodejs')
  )

  .map((resource: ISamFunction) => ({
    filename: resource.Properties.Handler.split('.')[0],
    entryPath: resource.Properties.CodeUri.split('/').splice(3).join('/'),
  }))

  .reduce(
    (resources, resource) =>
      Object.assign(resources, {
        [`${resource.filename}`]: `./src/${resource.entryPath}${resource.filename}.ts`,
      }),
    {}
  );

Matt Morgan • Jul 23 '20

Nice and thanks for the reminder. Got to get around to trying this!

Matt Morgan • Jul 24 '20

After leaving the same silly comment twice, I finally got around to putting this in and it works great. Thanks so much for the input!

Matt Morgan • Jun 16 '20

Thanks so much for the comment. I'll definitely try this!

Nick Bordeau • Jul 21 '20

Great article! Whenever I try to use a "build" or "dist" folder I get the following errors:

Build Failed
Error: NodejsNpmBuilder:NpmPack - NPM Failed: npm ERR! code ENOLOCAL
npm ERR! Could not install from "/path/to/dist/folder" as it does not contain a package.json file.

I would love to learn about how you got around that!

Matt Morgan • Jul 22 '20

Hey Nick, thanks for reading! If you're getting that error, I assume you're doing a sam build? Using the technique in this article, you won't use sam build and the reasons for that are outlined in the article. Instead you npm run build and then sam local start-api or sam deploy. If that's what you're doing and you're still getting the error above, let me know and I'll try to figure it out. It's been a bit since I've run this.

Nick Bordeau • Jul 22 '20

How would this work if I were to try to add dependency layers?

Matt Morgan • Jul 22 '20

I haven't actually done that myself with webpack - since the point of webpack is to bundle and tree-shake all the dependencies - but if you are solid on creating a Lambda layer with your modules, you'd just need to set whatever dependencies are in the layer as an external. See webpack.js.org/configuration/exter...
A lot of people do this with aws-sdk since it's already available in Lambda, but I've seen benchmarks show that you can actually get slightly faster cold starts if you bundle it so that's why I didn't do that in this example (though I have done it and TBH haven't noticed any difference either way).
If already using webpack with SAM, I'd probably only worry about Lambda layers if A) I had some huge dependency or B) I had some layer I wanted to share across many projects.

Nick Bordeau • Jul 23 '20

Do you have an example of what that structure looks like? Right now, I package everything separately with Webpack and then build the application with sam build. Here's the basic repo structure I use: github.com/Borduhh/serverless-SAM-...

I'm curious to know if this might be a more efficient way to do it though.

Matt Morgan • Jul 23 '20

The main thing I'm going for is a single package.json at the root of my repo, not multiple throughout. I introduced lerna into a UI project last year and my team hated having to manage dependencies (i.e. update them!) at multiple layers throughout the project. We ended up pulling that out for a single webpack build that could produce multiple bundles from the entrypoints. It's much cleaner and builds much faster!
So my article is about applying those same principles to building multiple Lambda functions from the same code base without multiple package.json files.
Like I said, I haven't worked with Lambda layers because I just webpack my entire project. What's the use case for using layers in your app? Are you just trying to make the bundle smaller? Share something across several code repos?

Nick Bordeau • Jul 23 '20

Yeah, that makes a lot of sense. I am using layers for precisely that reason, to avoid having to go into each package and update shared dependencies (i.e. a lot of our functions will use the aws-sdk package so I have an AwsSdkLayer which loads that subset of tools). That way I can pop into the layer, and update the dependency once and I am done.

It sounds like this would be similar though in a more efficient manner. I am just am having trouble wrapping my head around what a larger project structure would look like with multiple functions that share the same code.

Matt Morgan • Jul 24 '20

The project structure ends up looking a lot like your typical Express or NestJS app. That's the great thing about this approach. So we might do something like
/aws
/handlers
/repositories
/utils

The handlers are like controllers or routes in your average NodeJS app and everything else is shared or at least potentially shared. If you are somewhat careful in your imports, you should be able tree-shake away the things you don't need when you build. I did have a developer point out a little leakage with some extra unnecessary code showing up in a function due to the way things were being imported, but it hardly added any size and our functions are around 5MB, well below the 50MB limit.

I don't really have any other insights around lambda layers except I haven't felt the need to use them except for adding binaries to a lambda runtime.

We also follow the practice of putting a unit test right next to each of the code modules. The tree-shaking approach helps there too since obviously none of the production code has imports from the tests.

Patrick Foley • Mar 15 '20

This was super useful - thanks!

I remember when create-react-app didn't have typescript and then after a while, it did. I hope and ultimately suspect that sam will adopt an approach something like yours in the future as well. Think it will?

Matt Morgan • Mar 16 '20

Glad you found this helpful, Patrick!

As for the future of sam build, I don't doubt that eventually we'll get more customization options, however the sam team has their work cut out for them in supporting node, dotnet, ruby, golang, python and java. The sam build process is a lot less skippable for some of those other languages as they all package dependencies in different ways.

I'll also say that while I love this tool, I find when I read through AWS docs and examples, that I have a philosophical difference. For example, CDK TypeScript examples that write lambda functions in vanilla JavaScript. If you're already writing TypeScript and have committed to a build process, why not gain all the benefits? Another thing I notice is multiple package.json files in the same project tree. Seems like unnecessary complexity and that's why I wanted to blog about simpler alternatives.

At the end, these problems can, will and are being solved by the community (and not just me) and my suspicion is that AWS is quite fine to leave it to us while they focus on the really hard problems like making linux faster.

Matt Morgan • Mar 16 '20

Just FYI, was discussing a related topic with a colleague and I came across this: github.com/awslabs/aws-sam-cli/blo...

This isn't implemented. You can find an issue for a design and an aborted PR on github, but they definitely plan to have a more flexible build system, most likely that will entail adding your own transpilation/etc step, at least to start.

OMID EIDIVANDI • Nov 5 '20

Hi i'm challenging with that in a serverless , i like to get working sam ;)

i have a structure as

configs/
cfn/
DoThis
func1.yml
DoThat
func1.yml
DataStoreProcess
dynamodb.yml
StorageProcess
s3.yml
ProcessMessageBus
SNS.yml
SQS.yml

src
/node
/netcore
infra
scripts

have you any idea about theses kind of Situ

OMID EIDIVANDI • Nov 7 '20 • Edited

finally i foud the way to handle all nested templates using this part of code

const templates = walkSync(join(__dirname, "../../cfn/"));
const AllEnteries: any[] = [];

templates.forEach((fl) => {
    const { Globals, Resources } = yamlParse(readFileSync(fl, "utf-8"));
    const GlobalFunction = Globals?.Function ?? {};
    if (Resources) {
        const entries = Object.values(Resources)
            .filter((resource: ISamFunction) => resource.Type === "AWS::Serverless::Function")
            .filter((resource: ISamFunction) => (resource.Properties?.Runtime ?? GlobalFunction.Runtime).startsWith("nodejs"))
            .map((resource: ISamFunction) => ({
                filename: resource.Properties.Handler.split(".")[0],
                entryPath: resource.Properties.CodeUri.split("/").join("/"),
                name: resource.Properties.CodeUri.split("/").pop()
            }))
            .reduce(
                (resources, resource) =>
                    Object.assign(resources, {
                        [`${resource.name}`]: `${handlerPath}/${resource.name}/${resource.filename}.ts`,
                    }),
                {},
            );

        AllEnteries.push(entries);
    }
});

const webpackEnteries = Object.assign({}, ...AllEnteries);

Erick • Dec 23 '21 • Edited

Great article! I'm coming from the J2EE world and this is a huge help on how to get a project up & going!

Quick question - how did you get the prettier output from webpack? Really like the pretty/organized look.

Matt Morgan • Dec 23 '21

Hi Erick, thanks for reading. I can't really claim any credit for that output. I think webpack 5 provides that out of the box. Did you have trouble seeing something like that in your own project?

Erick • Dec 23 '21

Yeah...what I posted is the current webpack. I'll just play around with it.

Thanks for the fast reply...if I find something neat I'll follow up here.

Matt Martz • Mar 9 '20

For your resolve extensions... you list them in order of .js and then .ts... (which is the order that webpack looks for them in)... shouldn't you have .ts first?

I guess if you have a module with both .js and .ts extensions you have other problems anyways... so maybe it doesn't matter?

Matt Morgan • Mar 9 '20

That would only matter if some directory had both js and ts, correct. I'm not supporting js because I think I might write some but because I'm webpacking node modules. In the unlikely event some module has both a js and ts file, I probably want to prioritize the js, as that's what the author intended.

Ayoub Korkot • Dec 13 '21 • Edited

Hello Matt Morgan
Thanks a lot for this great article.

I tried to use this boilerplate as a starter kit for one of our new serverless applications.

I used "webpack-node-externals" as an alternative to tell webpack to ignore node_modules libs, but unfortunately SAM does not recognize packages outside generated webpackes microservices when running "sam local start-api or start-lambda"....

After searching on the internet it seems there is no way to tell SAM to load externals node_modules folder when trying to simulate api / lambda locally...

Do you have any Idea about this issue ? Maybe a workaround for this please ?

Thanks in advance for your help!

Matt Morgan • Dec 13 '21

Hi Ayoub, working with node_modules externals is a bit odd in SAM. This blog post may help you understand how it works: aws.amazon.com/blogs/compute/worki...

I was able to get this project working by following that technique. You can see this branch: github.com/elthrasher/sam-typescri...

My preference is to bundle modules in my functions and not make them external, but if you want to do that, that should get you started. I really don't like having to repeat the package.json and if you have a lot of dependencies, that could really be a burden.

Also since writing this blog post, I've moved most of my bundling to esbuild.github.io/. Recommend you give that a look as well as it really speeds things along.