I've been spending some quality time with the Serverless Application Model from AWS. I'm really enjoying this tool and I see it as a fantastic way for developers to learn more about the AWS ecosystem while being productive. However, it doesn't natively support TypeScript, my language of choice, and the build system it ships with leaves something to be desired. I also wanted to come up with a way to share some code between functions without the complexity of publishing and installing a private npm package or using lambda layers. The built-in tree-shaking and minification that webpack provides are also very attractive.
tl;dr
Feel free to skip ahead to the repo.
Table of Contents
- AWS SAM
- Webpack
- Plugin
- Toolchain
- Project Structure
- Install Packages
- Hello Lambda
- Goodbye Lambda
- Unit Tests
- Webpack Config
- Another Dependency
- SAM Template
- Local Test
- Deploy
- Next Steps
- Update
AWS SAM
SAM is a cli tool that enables local development and also provides access to an extended version of CloudFormation. I have previously written about combining SAM with AWS Cloud Developer Kit. In this case I'm just going to be using SAM.
Webpack
Many of us have struggled with messy webpack configuration. Copy-pasted webpack files can be tough to handle, with weird plugins and orphaned config options. Believe me when I say it doesn't have to be that way. The best way to deal with webpack is to start with a simple configuration and only add the options and plugins that you need and be sure to comment anything that's unclear. This is a bit like all coding if you think about it, but for some reason we don't always apply the same standards to our config files that we do to our source code.
So if you come to this article hating webpack, just bear with me and maybe you'll learn to do as I do and only allow the configuration options that you like and understand!
Plugin
I should mention there already exists a webpack plugin that seeks to solve the same problem, just in a different way. I haven't used it myself, but I read through the code and seems like it parses your template.yaml file to figure out what the entry points should be. I was thinking about contrasting the plugin with my approach, but there's already a great post on the subject. Check out aws-sam-webpack-plugin. Both approaches look quite viable to me. Mine has fewer moving parts, but maybe just a touch more work involved - adding a new function means touching both your template.yaml
and your webpack.config.ts
files.
Toolchain
I'm running Node 12 and the latest (as of this writing) version of SAM. sam local
commands depend on having Docker installed as your function will be run in a containerized environment (and yeah that's ironic, but works very well).
% node --version
v12.18.2
% sam --version
SAM CLI, version 1.0.0
% docker --version
Docker version 19.03.8, build afacb8b
If you don't have these tools installed, see the guides for nodejs, SAM, and Docker.
Project Structure
The default project structure you get with sam init
will put a template.yaml
at the project root, then create a subdirectory complete with a package.json
for each function. If you sam build
, SAM will search your entire project structure for package.json
files, copy everything into .aws-sam/build
and npm install
for each function you've staked out in your project. There isn't any way to add bundling or transpilation to the build process.
If I want to run my tests, I need to cd into the function directory, install my test runner (jest, mocha, whatever) there and run the test from that directory. Additionally the sam build
step will be a bit slower as it it will run one npm install
operation for each of my functions. Since each of these functions will have its own package.json
file, that means each of them manages its dependencies independently and creates more work for me to keep my dependencies up-to-date.
My project will maintain a single package.json
file at the root. Whatever dependencies my functions need will be shared. Because webpack does automatic tree-shaking, I can have confidence that unneeded dependencies won't be included in my bundles.
I'm going to put my functions in ./src/handlers
and allow for the possibility of some shared modules under src
.
Install Packages
I'm going to install most of my packages as devDependencies
. In a traditional nodejs app, this would let me have a build process that eventually runs npm prune --production
to remove all the devDependencies to make my node_modules smaller. But because I'm using webpack and the twice-aforementioned tree-shaking, this is really just a convention and not meaningful to my build process. This convention will be useful for other developers to understand my code. Anything installed under dependencies
should be considered as a dependency of live code while devDependencies
should be seen as libraries that help me build, test and lint my application.
npm i -D @types/webpack ts-loader ts-node typescript webpack webpack-cli
I'm installing the typings for webpack because I will write my config in TypeScript and include the typing for Configuration
which will help me avoid configuration typos. I'm going to use the webpack cli (as opposed to programmatic) so I need to install both of those packages. I'll be coding in TypeScript, so I need that. ts-loader
is a webpack plugin that enables the transpilation of .ts
files into webpack bundles and ts-node
is needed to write my configuration file in TypeScript. None of these dependencies will wind up in my Lambda functions unless I explicitly import them!
Since I'm using TypeScript, I need a tsconfig.json
file.
{
"compilerOptions": {
"alwaysStrict": true,
"module": "commonjs",
"noImplicitAny": true,
"target": "es2019"
}
}
I could add a few more compiler options, but let's keep it simple for now. It's important that I've set module
to commonjs
in order for webpack to understand module loading. More on that here. I've set the target to es2019 as that version of ECMAScript is fully implemented in NodeJS 12. If I choose an earlier compilation target, TypeScript will shim functionality that wasn't available in that version of ECMAScript and my bundle will be larger (for no good reason).
I'm also going to install eslint to further help me avoid errors and keep my code as clean as possible. I wrote about the use of eslint in another article and I'm doing exactly the same thing here. I'll round out the devDependencies with rimraf
because it's good to clean before you build.
Hello Lambda
I'm not improving on the sample project much here. My function is just going to say "Hello." I do want to install @types/aws-lambda
so I can import the expected return type and make sure I'm doing something that Lambda expects.
import { APIGatewayProxyResult } from 'aws-lambda';
export const handler = async (): Promise<APIGatewayProxyResult> => {
return { body: JSON.stringify({ message: 'hello' }), statusCode: 200 };
};
Even though my function doesn't actually do anything asynchronous, I need to make it an async function. Otherwise Lambda will expect a callback.
Goodbye Lambda
The same thing here. I'm putting these functions in ./src/handlers/hello.ts
and ./src/handlers/goodbye.ts
.
import { APIGatewayProxyResult } from 'aws-lambda';
export const handler = async (): Promise<APIGatewayProxyResult> => {
return { body: JSON.stringify({ message: 'goodbye' }), statusCode: 200 };
};
Unit Tests
A great thing about writing Lambda Functions is they are very easy to test. I'm going to use jest
and ts-jest
. I simply call the function and test the output. Here I'm using jest snapshots because they make testing very easy.
import { handler } from './hello';
describe('hello handler', () => {
it('should say hello', async () => {
const result = await handler();
expect(result).toMatchSnapshot();
});
});
Webpack Config
I will create a webpack.config.ts
file in the root of my project. Webpack is smart enough to realize that it should use that file by default and will employ ts-node
to parse it, if installed (otherwise it's unlikely to work). This file just needs to export a configuration object.
import { resolve } from 'path';
import { Configuration } from 'webpack';
const config: Configuration = {
// some stuff in here
};
export default config;
Okay, now to just get the important config bits. First of all, I'm going to be exporting two Lambda functions so I need two separate bundles. The way webpack builds multiple bundles is by specifying multiple entry points.
const config: Configuration = {
entry: { hello: './src/handlers/hello.ts', goodbye: './src/handlers/goodbye.ts' },
output: {
filename: '[name].js',
libraryTarget: 'commonjs2',
path: resolve(__dirname, 'build'),
},
// more
};
This identifies my entry points as hello
and goodbye
and specifies the typescript files that will provide the source for those bundles. My output will sub in the name of the entry using commonjs2 and put everything in the build directory leaving me with ./build/hello.js
and ./build/goodbye.js
.
But this won't work yet. I need a few more config elements to bring it all together.
const config: Configuration = {
// more
module: {
rules: [{ test: /\.ts$/, loader: 'ts-loader' }],
},
resolve: {
extensions: ['.js', '.ts'],
},
target: 'node',
// more
};
ts-loader
is need here so that my TypeScript files get transpiled with tsc
. I need to specify extensions for module resolution so that webpack knows import foo from './foo';
might get me foo.js
or foo.ts
. Even though I'm going to write all of my code in TypeScript, it's likely I'll want to import a dependency that's either written in or transpiled to JavaScript, so I need the .js
extension as well. Targeting node
lets webpack know that nodejs apis such as fs
are available.
Lastly I'm going to give myself a way to run webpack in non-prod mode if I feel like it.
const config: Configuration = {
// more
mode: process.env.NODE_ENV === 'dev' ? 'development' : 'production',
// more
};
If I set NODE_ENV
to dev
(for example NODE_ENV=dev npm run build
) then my code won't be minified - something that might help with debugging. Ultimately I might like to set up a workflow where I always run my functions in dev mode while developing and then they are tested in production mode before being deployed. I also might want to look at adding webpack devtools as my project grows in complexity.
My build system is complete! Now I can run npx webpack
and find goodbye.js
and hello.js
in my build
directory. I'll add scripts to my package.json
for ease.
% npm run build
> sam-typescript-webpack@0.0.1 build /Users/mattmorgan/mine/sam-typescript-webpack
> npm run clean && webpack
> sam-typescript-webpack@0.0.1 clean /Users/mattmorgan/mine/sam-typescript-webpack
> rimraf build
Hash: 5189a33d1ad6129ac421
Version: webpack 4.41.5
Time: 932ms
Built at: 02/10/2020 7:04:58 AM
Asset Size Chunks Chunk Names
goodbye.js 1.07 KiB 0 [emitted] goodbye
hello.js 1.07 KiB 1 [emitted] hello
Entrypoint hello = hello.js
Entrypoint goodbye = goodbye.js
[0] ./src/handlers/hello.ts 188 bytes {1} [built]
[1] ./src/handlers/goodbye.ts 190 bytes {0} [built]
Another Dependency
Now I'll show what happens to my bundles when I introduce another dependency. Imagine I wanted to use winston for logging (actually console works pretty well in Lamba but perhaps I have a good reason). I'll npm i winston
(my first real dependency), dash off a quick logging util and then add it to one of my handlers.
import { createLogger, format, transports } from 'winston';
const logger = createLogger({
level: 'info',
format: format.json(),
transports: [new transports.Console()],
});
export default logger;
import { APIGatewayProxyResult } from 'aws-lambda';
import logger from '../util/logger';
export const handler = async (): Promise<APIGatewayProxyResult> => {
logger.info(`time to say 'hello'`);
return { body: JSON.stringify({ message: 'hello' }), statusCode: 200 };
};
I've added this to hello.ts
but not goodbye.ts
. When I build again, only the hello bundle has increased in size.
% npm run build
> sam-typescript-webpack@0.0.1 build /Users/mattmorgan/mine/sam-typescript-webpack
> npm run clean && webpack
> sam-typescript-webpack@0.0.1 clean /Users/mattmorgan/mine/sam-typescript-webpack
> rimraf build
Hash: cd9e2e299faff34c0ed6
Version: webpack 4.41.5
Time: 1224ms
Built at: 02/10/2020 7:09:16 AM
Asset Size Chunks Chunk Names
goodbye.js 1.07 KiB 0 [emitted] goodbye
hello.js 195 KiB 1 [emitted] hello
Entrypoint hello = hello.js
Entrypoint goodbye = goodbye.js
[3] external "util" 42 bytes {1} [built]
[7] external "os" 42 bytes {1} [built]
[17] external "stream" 42 bytes {1} [built]
[74] external "fs" 42 bytes {1} [built]
[101] ./src/handlers/hello.ts 282 bytes {1} [built]
[102] ./src/util/logger.ts 295 bytes {1} [built]
[118] ./node_modules/logform sync ^\.\/.*\.js$ 1.09 KiB {1} [built]
[120] ./node_modules/logform/dist sync ^\.\/.*\.js$ 508 bytes {1} [built]
[138] external "path" 42 bytes {1} [built]
[168] external "zlib" 42 bytes {1} [built]
[187] external "tty" 42 bytes {1} [built]
[189] external "string_decoder" 42 bytes {1} [built]
[191] external "http" 42 bytes {1} [built]
[192] external "https" 42 bytes {1} [built]
[200] ./src/handlers/goodbye.ts 190 bytes {0} [built]
+ 186 hidden modules
hello.js
is now 195kb while goodbye.js
remains just over 1kb. Console is looking pretty good at this point. When writing Lambda functions, it's advisable to minimize imported dependencies. In fact some node dependencies (as opposed to browser app dependencies) don't minify very well. You may find yourself needing to separate your code and dependencies using something like webpack node externals to get around this problem.
Also note in my output above all the dependencies marked external
. This is because I set target: node
in my webpack config. It means webpack is assuming these libraries will simply be available in my target environment
SAM Template
The SAM templating engine is an extension of CloudFormation. SAM templates can refer to both SAM resources and CloudFormation resources. When deployed, SAM will create a CloudFormation stack, a group of resources created in your AWS account. Read more about SAM and CloudFormation.
SAM (and CloudFormation) templates are written in yaml. The cli tool has a built-in validator so you'll never need to worry about deploying invalid yaml. There's even a sam validate
command which is run automatically with sam deploy
.
% sam validate
/Users/mattmorgan/mine/sam-typescript-webpack/template.yaml is a valid SAM Template
Now as for the template itself, every SAM template will begin with something that looks like this:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sam-typescript-webpack
Sample SAM Template for sam-typescript-webpack
AWS versions its specifications by the date it was published. These template versions don't change often and AWS is known for supporting backward compatibility.
Globals:
Function:
CodeUri: build/
Runtime: nodejs12.x
Timeout: 300
Globals are a feature unique to SAM that let me write a slightly more DRY config file and work as you'd expect. You can override any global at the resource level.
In this case, I'm specifying that all my code will come from the build directory, setting my node version and function timeout (5 minutes).
Resources:
HelloFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
Handler: hello.handler
Events:
Hello:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
GoodbyeFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
Handler: goodbye.handler
Events:
Goodbye:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /goodbye
Method: get
I'm defining my functions here. hello.handler
means that inside the CodeUri
location (specified in Globals), there will be a hello.js
that exposes a handler
function.
I don't have to explicitly declare the ApiGateway service as a Resource. Setting Type: Api
on my Function is enough. I might prefer to use HttpApi, but it is still in beta and is not yet supported in sam local
runs.
Finally I'll define some Outputs:
Outputs:
# ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
# Find out more about other implicit resources you can reference within SAM
# https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
HelloApi:
Description: 'API Gateway endpoint URL for Prod stage for Hello function'
Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/'
GoodbyeApi:
Description: 'API Gateway endpoint URL for Prod stage for Goodbye function'
Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/goodbye/'
In this case, I'm outputting the URLs that ApiGateway assigned to my functions so I can start testing.
Local Test
sam local
can be used to invoke a Lambda function and test different payloads and triggers. In this case I will use start-api
to run my Functions locally with an Api Gateway.
% sam local start-api
Mounting GoodbyeFunction at http://127.0.0.1:3000/goodbye [GET]
Mounting HelloFunction at http://127.0.0.1:3000/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2020-02-09 22:57:15 * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
I navigate to http://127.0.0.1:3000/hello
and see {"message":"hello"}
. Easy stuff.
Deploy
The first time I deploy, it's a good idea to use sam deploy --guided
. I of course need credentials for my AWS account to do this.
% sam deploy --guided
Configuring SAM deploy
======================
Looking for samconfig.toml : Not found
Setting default arguments for 'sam deploy'
=========================================
Stack Name [sam-app]: sam-typescript-webpack
AWS Region [us-east-1]:
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
Confirm changes before deploy [y/N]:
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]:
Save arguments to samconfig.toml [Y/n]:
Looking for resources needed for deployment: Found!
Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-16ecthy96ctvq
A different default S3 bucket can be set in samconfig.toml
Saved arguments to config file
Running 'sam deploy' for future deployments will use the parameters saved above.
The above parameters can be changed by modifying samconfig.toml
Learn more about samconfig.toml syntax at
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
Deploying with following values
===============================
Stack name : sam-typescript-webpack
Region : us-east-1
Confirm changeset : False
Deployment s3 bucket : aws-sam-cli-managed-default-samclisourcebucket-16ecthy96ctvq
Capabilities : ["CAPABILITY_IAM"]
Parameter overrides : {}
Initiating deployment
=====================
Uploading to sam-typescript-webpack/29e43b24a388e6fc03dfca21d52effe2 57738 / 57738.0 (100.00%)
Uploading to sam-typescript-webpack/1d786b318a187cf6c9952adbe105dfb8.template 1387 / 1387.0 (100.00%)
Waiting for changeset to be created..
CloudFormation stack changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation LogicalResourceId ResourceType
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add GoodbyeFunctionGoodbyePermissionProd AWS::Lambda::Permission
+ Add GoodbyeFunctionRole AWS::IAM::Role
+ Add GoodbyeFunction AWS::Lambda::Function
+ Add HelloFunctionHelloPermissionProd AWS::Lambda::Permission
+ Add HelloFunctionRole AWS::IAM::Role
+ Add HelloFunction AWS::Lambda::Function
+ Add ServerlessRestApiDeploymenteca00d9d3f AWS::ApiGateway::Deployment
+ Add ServerlessRestApiProdStage AWS::ApiGateway::Stage
+ Add ServerlessRestApi AWS::ApiGateway::RestApi
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Changeset created successfully. arn:aws:cloudformation:us-east-1:336848621206:changeSet/samcli-deploy1581337063/c7e95df9-bf73-4360-ab9f-9ce538bac740
2020-02-10 07:17:49 - Waiting for stack create/update to complete
CloudFormation events from changeset
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus ResourceType LogicalResourceId ResourceStatusReason
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS AWS::IAM::Role GoodbyeFunctionRole Resource creation Initiated
CREATE_IN_PROGRESS AWS::IAM::Role HelloFunctionRole Resource creation Initiated
CREATE_IN_PROGRESS AWS::IAM::Role GoodbyeFunctionRole -
CREATE_IN_PROGRESS AWS::IAM::Role HelloFunctionRole -
CREATE_COMPLETE AWS::IAM::Role GoodbyeFunctionRole -
CREATE_COMPLETE AWS::IAM::Role HelloFunctionRole -
CREATE_IN_PROGRESS AWS::Lambda::Function HelloFunction -
CREATE_IN_PROGRESS AWS::Lambda::Function HelloFunction Resource creation Initiated
CREATE_IN_PROGRESS AWS::Lambda::Function GoodbyeFunction Resource creation Initiated
CREATE_COMPLETE AWS::Lambda::Function HelloFunction -
CREATE_IN_PROGRESS AWS::Lambda::Function GoodbyeFunction -
CREATE_COMPLETE AWS::Lambda::Function GoodbyeFunction -
CREATE_IN_PROGRESS AWS::ApiGateway::RestApi ServerlessRestApi Resource creation Initiated
CREATE_IN_PROGRESS AWS::ApiGateway::RestApi ServerlessRestApi -
CREATE_COMPLETE AWS::ApiGateway::RestApi ServerlessRestApi -
CREATE_IN_PROGRESS AWS::ApiGateway::Deployment ServerlessRestApiDeploymenteca00d9d3f -
CREATE_IN_PROGRESS AWS::Lambda::Permission HelloFunctionHelloPermissionProd -
CREATE_IN_PROGRESS AWS::Lambda::Permission GoodbyeFunctionGoodbyePermissionProd -
CREATE_IN_PROGRESS AWS::ApiGateway::Deployment ServerlessRestApiDeploymenteca00d9d3f Resource creation Initiated
CREATE_IN_PROGRESS AWS::Lambda::Permission HelloFunctionHelloPermissionProd Resource creation Initiated
CREATE_IN_PROGRESS AWS::Lambda::Permission GoodbyeFunctionGoodbyePermissionProd Resource creation Initiated
CREATE_COMPLETE AWS::ApiGateway::Deployment ServerlessRestApiDeploymenteca00d9d3f -
CREATE_IN_PROGRESS AWS::ApiGateway::Stage ServerlessRestApiProdStage -
CREATE_IN_PROGRESS AWS::ApiGateway::Stage ServerlessRestApiProdStage Resource creation Initiated
CREATE_COMPLETE AWS::ApiGateway::Stage ServerlessRestApiProdStage -
CREATE_COMPLETE AWS::Lambda::Permission HelloFunctionHelloPermissionProd -
CREATE_COMPLETE AWS::Lambda::Permission GoodbyeFunctionGoodbyePermissionProd -
CREATE_COMPLETE AWS::CloudFormation::Stack sam-typescript-webpack -
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Stack sam-typescript-webpack outputs:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OutputKey-Description OutputValue
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
HelloApi - API Gateway endpoint URL for Prod stage for Hello function https://w32i72j3pc.execute-api.us-east-1.amazonaws.com/Prod/hello/
GoodbyeApi - API Gateway endpoint URL for Prod stage for Goodbye function https://w32i72j3pc.execute-api.us-east-1.amazonaws.com/Prod/goodbye/
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Successfully created/updated stack - sam-typescript-webpack in us-east-1
Everything looks good! I can navigate to the urls about in my browser and then find output in CloudWatch.
Next Steps
SAM is a very complete tool when it comes to building out new serverless applications, but some additional work is required if you want to connect to an existing RDS or enhance an existing local environment with serverless functions. However this is possible, so maybe I'll go through that in another post.
Update
Thanks to some very helpful feedback from @sandinosaso and @borduhh in the comments, the entries are determined dynamically by reading the SAM template. It could make a lot of sense to follow this pattern if you're putting a lot of functions in a single stack. If you aren't doing that many, hardcoding could still make sense.
Cover Image - St. Brendan's ship on the back of a whale, and his men praying, in Honorius Philoponus' Nova typis transacta navigation 1621; image from Sea Monsters on Medieval and Renaissance Maps by Chet Van Duzer
Top comments (25)
Thanks for sharing this is a great post!!.
Regarding your concern about your approach "adding a new function means touching both your template.yaml and your webpack.config.ts files." someone else figured out how to keep them in sync:
Here is the full config: gist.github.com/henrikbjorn/d2eef5...
I think combining both approaches is a great toolchain to have.
Thank you.
Here's the Typescript version as well.
Nice and thanks for the reminder. Got to get around to trying this!
After leaving the same silly comment twice, I finally got around to putting this in and it works great. Thanks so much for the input!
Thanks so much for the comment. I'll definitely try this!
Great article! Whenever I try to use a "build" or "dist" folder I get the following errors:
I would love to learn about how you got around that!
Hey Nick, thanks for reading! If you're getting that error, I assume you're doing a
sam build
? Using the technique in this article, you won't usesam build
and the reasons for that are outlined in the article. Instead younpm run build
and thensam local start-api
orsam deploy
. If that's what you're doing and you're still getting the error above, let me know and I'll try to figure it out. It's been a bit since I've run this.How would this work if I were to try to add dependency layers?
I haven't actually done that myself with webpack - since the point of webpack is to bundle and tree-shake all the dependencies - but if you are solid on creating a Lambda layer with your modules, you'd just need to set whatever dependencies are in the layer as an external. See webpack.js.org/configuration/exter...
A lot of people do this with aws-sdk since it's already available in Lambda, but I've seen benchmarks show that you can actually get slightly faster cold starts if you bundle it so that's why I didn't do that in this example (though I have done it and TBH haven't noticed any difference either way).
If already using webpack with SAM, I'd probably only worry about Lambda layers if A) I had some huge dependency or B) I had some layer I wanted to share across many projects.
Do you have an example of what that structure looks like? Right now, I package everything separately with Webpack and then build the application with
sam build
. Here's the basic repo structure I use: github.com/Borduhh/serverless-SAM-...I'm curious to know if this might be a more efficient way to do it though.
The main thing I'm going for is a single package.json at the root of my repo, not multiple throughout. I introduced lerna into a UI project last year and my team hated having to manage dependencies (i.e. update them!) at multiple layers throughout the project. We ended up pulling that out for a single webpack build that could produce multiple bundles from the entrypoints. It's much cleaner and builds much faster!
So my article is about applying those same principles to building multiple Lambda functions from the same code base without multiple package.json files.
Like I said, I haven't worked with Lambda layers because I just webpack my entire project. What's the use case for using layers in your app? Are you just trying to make the bundle smaller? Share something across several code repos?
Yeah, that makes a lot of sense. I am using layers for precisely that reason, to avoid having to go into each package and update shared dependencies (i.e. a lot of our functions will use the
aws-sdk
package so I have anAwsSdkLayer
which loads that subset of tools). That way I can pop into the layer, and update the dependency once and I am done.It sounds like this would be similar though in a more efficient manner. I am just am having trouble wrapping my head around what a larger project structure would look like with multiple functions that share the same code.
The project structure ends up looking a lot like your typical Express or NestJS app. That's the great thing about this approach. So we might do something like
/aws
/handlers
/repositories
/utils
The handlers are like controllers or routes in your average NodeJS app and everything else is shared or at least potentially shared. If you are somewhat careful in your imports, you should be able tree-shake away the things you don't need when you build. I did have a developer point out a little leakage with some extra unnecessary code showing up in a function due to the way things were being imported, but it hardly added any size and our functions are around 5MB, well below the 50MB limit.
I don't really have any other insights around lambda layers except I haven't felt the need to use them except for adding binaries to a lambda runtime.
We also follow the practice of putting a unit test right next to each of the code modules. The tree-shaking approach helps there too since obviously none of the production code has imports from the tests.
This was super useful - thanks!
I remember when create-react-app didn't have typescript and then after a while, it did. I hope and ultimately suspect that sam will adopt an approach something like yours in the future as well. Think it will?
Glad you found this helpful, Patrick!
As for the future of sam build, I don't doubt that eventually we'll get more customization options, however the sam team has their work cut out for them in supporting node, dotnet, ruby, golang, python and java. The sam build process is a lot less skippable for some of those other languages as they all package dependencies in different ways.
I'll also say that while I love this tool, I find when I read through AWS docs and examples, that I have a philosophical difference. For example, CDK TypeScript examples that write lambda functions in vanilla JavaScript. If you're already writing TypeScript and have committed to a build process, why not gain all the benefits? Another thing I notice is multiple package.json files in the same project tree. Seems like unnecessary complexity and that's why I wanted to blog about simpler alternatives.
At the end, these problems can, will and are being solved by the community (and not just me) and my suspicion is that AWS is quite fine to leave it to us while they focus on the really hard problems like making linux faster.
Just FYI, was discussing a related topic with a colleague and I came across this: github.com/awslabs/aws-sam-cli/blo...
This isn't implemented. You can find an issue for a design and an aborted PR on github, but they definitely plan to have a more flexible build system, most likely that will entail adding your own transpilation/etc step, at least to start.
Hi i'm challenging with that in a serverless , i like to get working sam ;)
i have a structure as
configs/
cfn/
DoThis
func1.yml
DoThat
func1.yml
DataStoreProcess
dynamodb.yml
StorageProcess
s3.yml
ProcessMessageBus
SNS.yml
SQS.yml
src
/node
/netcore
infra
scripts
have you any idea about theses kind of Situ
finally i foud the way to handle all nested templates using this part of code
Great article! I'm coming from the J2EE world and this is a huge help on how to get a project up & going!
Quick question - how did you get the prettier output from webpack? Really like the pretty/organized look.
Hi Erick, thanks for reading. I can't really claim any credit for that output. I think webpack 5 provides that out of the box. Did you have trouble seeing something like that in your own project?
Yeah...what I posted is the current webpack. I'll just play around with it.
Thanks for the fast reply...if I find something neat I'll follow up here.
For your resolve extensions... you list them in order of .js and then .ts... (which is the order that webpack looks for them in)... shouldn't you have .ts first?
I guess if you have a module with both .js and .ts extensions you have other problems anyways... so maybe it doesn't matter?
That would only matter if some directory had both js and ts, correct. I'm not supporting js because I think I might write some but because I'm webpacking node modules. In the unlikely event some module has both a js and ts file, I probably want to prioritize the js, as that's what the author intended.
Hello Matt Morgan
Thanks a lot for this great article.
I tried to use this boilerplate as a starter kit for one of our new serverless applications.
I used "webpack-node-externals" as an alternative to tell webpack to ignore node_modules libs, but unfortunately SAM does not recognize packages outside generated webpackes microservices when running "sam local start-api or start-lambda"....
After searching on the internet it seems there is no way to tell SAM to load externals node_modules folder when trying to simulate api / lambda locally...
Do you have any Idea about this issue ? Maybe a workaround for this please ?
Thanks in advance for your help!
Hi Ayoub, working with node_modules externals is a bit odd in SAM. This blog post may help you understand how it works: aws.amazon.com/blogs/compute/worki...
I was able to get this project working by following that technique. You can see this branch: github.com/elthrasher/sam-typescri...
My preference is to bundle modules in my functions and not make them external, but if you want to do that, that should get you started. I really don't like having to repeat the package.json and if you have a lot of dependencies, that could really be a burden.
Also since writing this blog post, I've moved most of my bundling to esbuild.github.io/. Recommend you give that a look as well as it really speeds things along.