In my previous post I talked a little bit about how you can get started with running a puppeteer script in an AWS Lambda function. Using the puppeteer-core
and chrome-aws-lambda
libraries, you can get your code down to a size that will fit into the 50MB limit.
This is a relatively good solution, that will work for most cases. However, there are some drawbacks:
- If your code becomes a bit more complex you might end up exceeding the limit.
- You might have multiple functions that require puppeteer.
- Every time you make a change to some of your code, you have to re-upload everything to AWS. This can take some time as these libraries are still more than 40MB in size.
Lucky for us, there is one solution to all 3 of these problems, we use layers. In this post, I will take you through extracting the two libraries required to run puppeteer, into an AWS Lambda layer and using this layer inside your function.
Creating your layer
The puppeteer-core
library is dependent on the chrome-aws-lambda
library as that is what installs the chromium bin file that is needed. This is why we move both of them into one layer. I found that even though our Lambda function can interact with the libraries inside a layer as if they were in its own node_modules
folder, the libraries themselves do not interact with each other in the same way. Which means that leaving puppeteer-core
in our Lambda function and only moving chrome-aws-lambda
will not work. (This was discovered by trial and error 😅).
The easiest way to do this is to create a new folder for your layer and add a package.json
with the two libraries as dependencies. Since these are node libraries, AWS requires them to be inside a folder called nodejs
, so our folder structure would be layer/nodejs/package.json
and the file would look something like this:
{
"name": "puppeteer-layer",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"create-layer": "pushd ../../ && rm -rf layer.zip && popd && npm i && cd ../ && zip -r ../layer.zip *"
},
"author": "",
"license": "ISC",
"dependencies": {
"chrome-aws-lambda": "^5.3.1",
"puppeteer-core": "^5.3.1"
}
}
We have one script in our package.json
file, it will create the zip file that we will upload to our layer in AWS. We now run npm run create-layer
and upload our file using the AWS management console. Go to the console and log in, select Lambda, from the side menu select Layers, click on Create layer fill in all of the details and upload the zip file.
Another note on running locally: Once you've moved these dependencies out of your Lambda function's
package.json
you might struggle to run this locally. Putting them into the dev dependencies is the easiest way to get around this problem.
Using your layer
The nice thing here is that none of your code has to change, just remember to remove those dependencies (or make them dev dependencies). Upload your new Lambda function without the dependency on puppeteer-core
and chrome-aws-lambda
and then we can tell it about the layer. In the console, select your function, and select Layers, you will see an empty list of layers. Click on Add a layer, select the Custom layers option and your newly created layer should appear in that list. Once added, everything should work as it did before.
Note: One last thing to keep in mind is that a layer is also limited to 50MB size, which means that you will not be able to move all of your dependencies to a single layer if they exceed this size. However, you can create and use more than one layer in your Lambda function.
Conclusion
Now that we have a layer we can easily create multiple functions that use these libraries and not worry about the size of our own code. We can also update our code much easier, since we would have a much smaller zip file to work with.
Top comments (6)
Hi, I've done the same and it's complaining about the size 51mb. Tried with same versions. Is there any trick or workaround?
As an alternative I'am using this layers github.com/shelfio/chrome-aws-lamb...
Hi there, I'm not really sure why you are seeing this problem. It maybe related to the compression you are using. I am using the default mac OS
zip
command and when I run the script above I get a zip file that is 47MB in size.The alternative also looks like a good option, I see they use Brotli to compress it.
I'm using same file compression ¯_(ツ)_/¯
Maybe because I'm using case sensitive APFS.
For now I'll stick with shelfio alternative
Great post, It really help me!
The syntax of the command is incorrect.
Should this be a different command for windows
I think the command would be quite different in Windows since most parts of it are unix based commands. Unfortunately I don't have a Windows machine to try it on. If you do figure it out it would be great if you posted it here for others that might need it.