Troubleshooting binaries in AWS Lambda

I promised to write an entire article on this topic after struggling for a couple hours to get the graphviz library working in AWS Lambda. In this article I wanted to cover the tools I used to create a working version of the binary.

Build target

If you're looking to compile a binary for your Lambda you could either run an EC2 instance with the Amazon Linux 2 AMI or use the amazonlinux:2 docker image. Here's how you can get an interactive shell running:

$ docker run --rm -it amazonlinux:2 bash

In either case, have in mind these won't ship compilation tools by default so you will have to install them. sudo yum groupinstall "Development Tools" would get you a decent list to start with. To find out about the shared dependencies for a binary you could use ldd.
Once you compiled your binary you should copy it to the development environment for triaging.

Development environment

When working on an AWS Lambda function with a custom binary it's definitely useful to have a fast feedback loop. You can run a container with docker-lambda for this and it will replicate filesystem permissions and binaries bundled by default (among other things).
If you spawn commonly used binaries, like /usr/bin/find, make sure these are bundled by default (if they're not, you could include them yourself):

$ docker run --rm --entrypoint /bin/bash lambci/lambda:provided -c 'ls -lha /usr/bin'

It works on my machine

Once you have the binaries and libraries necessary you can include them in your Lambda deployment or in case you want to re-use them you can upload it as a layer. It won't really matter the flavor of serverless of your choice.

Layers are mounted in the /opt directory so some mangling is required to run these custom binaries. Supposing your layer included the bin and lib directories you will need to append these values to the $PATH and $LD_LIBRARY environment variables respectively. Since this is needed at runtime, it will be only language-specific step required. Here's how you could do it in Node.js:

process.env.PATH = `${process.env.PATH}:/opt/bin`;
process.env.LD_LIBRARY_PATH = `${process.env.LD_LIBRARY_PATH}:/opt/lib`;
module.exports.handler= async () => {
  /* code */
}

For a list of environment variables, including runtime-specific ones, see here.

Layer tooling

No matter which serverless flavour you're using if you want to download a layer from an ARN you will need to use AWS CLI. Even in some cases, when you're still browsing alternatives for a layer, you want to check for its content first before downloading it. It could be to verify the binary it's adding or to check the size of the files it contains.
I've built a small tool that can help to check binaries included in a layer, it's size and permissions: describeawslayer.com.

Convenience kills reproducibility

This is kinda the motto behind DockerHub's registry success, right? I know AWS SAR exists but it's not necessarily targeted for dependencies. It feels like there's still room for a layer manager (or layer registry?) to fill in the gap of re-using existing layers. I'm aware that there's also awesome-lambda-layers but it falls short to display if there's active usage on those layers or to even mention an ARN for all listed items. I trust that a tool like this could speed up the development lifecycle and would expand the notion of what we think it's suited for serverless even further.

If you used a different tool set for troubleshooting your binary within AWS Lambda or if you encountered different problems I would love to hear from you in the comments section.

Other resources:

• Official documentation for AWS Lambda layers
• How do I create a Lambda layer using a simulated Lambda environment with Docker?
• How do I use Amazon Linux AMI native binary packages in an AWS Lambda deployment package?