DEV Community

loading...
Cover image for Puppeteer in an AWS Lambda Function Part 2

Puppeteer in an AWS Lambda Function Part 2

Gergana Young
Software developer, adventurer, and Star Wars geek extraordinaire.
・3 min read

In my previous post I talked a little bit about how you can get started with running a puppeteer script in an AWS Lambda function. Using the puppeteer-core and chrome-aws-lambda libraries, you can get your code down to a size that will fit into the 50MB limit.

This is a relatively good solution, that will work for most cases. However, there are some drawbacks:

  • If your code becomes a bit more complex you might end up exceeding the limit.
  • You might have multiple functions that require puppeteer.
  • Every time you make a change to some of your code, you have to re-upload everything to AWS. This can take some time as these libraries are still more than 40MB in size.

Lucky for us, there is one solution to all 3 of these problems, we use layers. In this post, I will take you through extracting the two libraries required to run puppeteer, into an AWS Lambda layer and using this layer inside your function.

Creating your layer

The puppeteer-core library is dependent on the chrome-aws-lambda library as that is what installs the chromium bin file that is needed. This is why we move both of them into one layer. I found that even though our Lambda function can interact with the libraries inside a layer as if they were in its own node_modules folder, the libraries themselves do not interact with each other in the same way. Which means that leaving puppeteer-core in our Lambda function and only moving chrome-aws-lambda will not work. (This was discovered by trial and error 😅).

The easiest way to do this is to create a new folder for your layer and add a package.json with the two libraries as dependencies. Since these are node libraries, AWS requires them to be inside a folder called nodejs, so our folder structure would be layer/nodejs/package.json and the file would look something like this:

{
  "name": "puppeteer-layer",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "create-layer": "pushd ../../ && rm -rf layer.zip && popd && npm i && cd ../ && zip -r ../layer.zip *"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "chrome-aws-lambda": "^5.3.1",
    "puppeteer-core": "^5.3.1"
  }
}
Enter fullscreen mode Exit fullscreen mode

We have one script in our package.json file, it will create the zip file that we will upload to our layer in AWS. We now run npm run create-layer and upload our file using the AWS management console. Go to the console and log in, select Lambda, from the side menu select Layers, click on Create layer fill in all of the details and upload the zip file.

Another note on running locally: Once you've moved these dependencies out of your Lambda function's package.json you might struggle to run this locally. Putting them into the dev dependencies is the easiest way to get around this problem.

Using your layer

The nice thing here is that none of your code has to change, just remember to remove those dependencies (or make them dev dependencies). Upload your new Lambda function without the dependency on puppeteer-core and chrome-aws-lambda and then we can tell it about the layer. In the console, select your function, and select Layers, you will see an empty list of layers. Click on Add a layer, select the Custom layers option and your newly created layer should appear in that list. Once added, everything should work as it did before.

Note: One last thing to keep in mind is that a layer is also limited to 50MB size, which means that you will not be able to move all of your dependencies to a single layer if they exceed this size. However, you can create and use more than one layer in your Lambda function.

Conclusion

Now that we have a layer we can easily create multiple functions that use these libraries and not worry about the size of our own code. We can also update our code much easier, since we would have a much smaller zip file to work with.

Discussion (6)

Collapse
vieraleonel profile image
Viera Leonel • Edited

Hi, I've done the same and it's complaining about the size 51mb. Tried with same versions. Is there any trick or workaround?

As an alternative I'am using this layers github.com/shelfio/chrome-aws-lamb...

Collapse
gerybbg profile image
Gergana Young Author

Hi there, I'm not really sure why you are seeing this problem. It maybe related to the compression you are using. I am using the default mac OS zip command and when I run the script above I get a zip file that is 47MB in size.

The alternative also looks like a good option, I see they use Brotli to compress it.

Collapse
vieraleonel profile image
Viera Leonel

I'm using same file compression ¯_(ツ)_/¯

Maybe because I'm using case sensitive APFS.

For now I'll stick with shelfio alternative

Great post, It really help me!

Collapse
akash8663 profile image
Abrar.Akash • Edited

The syntax of the command is incorrect.

    "create-layer": "pushd ../../ && rm -rf layer.zip && popd && npm i && cd ../ && zip -r ../layer.zip *"
Enter fullscreen mode Exit fullscreen mode
Collapse
akash8663 profile image
Abrar.Akash

Should this be a different command for windows

Collapse
gerybbg profile image
Gergana Young Author

I think the command would be quite different in Windows since most parts of it are unix based commands. Unfortunately I don't have a Windows machine to try it on. If you do figure it out it would be great if you posted it here for others that might need it.