How to Create a AWS Lambda Layer of Your Gemfile / Ruby Gem Dependencies

If you use rooling like AWS SAM or Serverless Framework to build and deploy your serverless Ruby functions, they'll take care of building your dependencies on a function by function basis and package them up for deployment. Great! Use them!

But if you prefer to have your own hands on the controls at all times, using gems with native dependencies (e.g. Nokogiri) can be a nightmare! If you prefer to travel light (or, maybe, like me you like coding adhoc functions in the AWS console), you can instead build your dependencies into a Lambda Layer and then use that layer across whatever Lambda functions you like without worrying about dependencies again.

But how? After an hour of trial and error plus a few tips from this neat article (though that one still used SAM), I figured out the magic spell required to locally build Ruby gems and their dependencies in a local Amazon Linux container and turn them into a layer.

Note: This post assumes you're familiar with the aws client and have the credentials to use Lambda with it. If not, this is too advanced for now. Google the aws CLI client and how to create an IAM user with the right permissions.

The code

Without further ado, here's the magic:

LAYER_NAME="my-ruby-layer"
mkdir $LAYER_NAME && cd $_

bundle init
bundle add http --skip-install
bundle add nokogiri --skip-install
rm Gemfile.lock

docker run --rm -v $PWD:/var/layer \
           -w /var/layer \
           amazon/aws-sam-cli-build-image-ruby2.7 \
           bundle install --path=ruby

mv ruby/ruby ruby/gems
zip -r layer.zip ruby

aws lambda publish-layer-version \
           --layer-name $LAYER_NAME \
           --region eu-west-1 \
           --compatible-runtimes ruby2.7 \
           --zip-file fileb://layer.zip

This results in a LayerVersionArn you can use with your Lambda functions. Once you've done this, loading the gems you need in the usual way (e.g. require 'nokogiri') will Just Work™.

If you use the AWS console, it'll let you pick this from a drop down menu which is how I like to do it:

How the code works

Here's a quick break down of what the code above does. First, we create a folder with the name of our layer:

LAYER_NAME="my-ruby-layer"
mkdir $LAYER_NAME && cd $_

Next, we create a Gemfile and add some gems to it. If you already have a Gemfile, copy it in at this point instead of doing this:

bundle init
bundle add http --skip-install
bundle add nokogiri --skip-install
rm Gemfile.lock

Note: I delete Gemfile.lock because the version of Bundler in the Lambda container clashes with mine.

Let's now instruct Docker to run the image Amazon provides for building Ruby 2.7 dependencies and do the bundle install there (while saving it to our normal filesystem):

docker run --rm -v $PWD:/var/layer \
           -w /var/layer \
           amazon/aws-sam-cli-build-image-ruby2.7 \
           bundle install --path=ruby

A quick directory name tweak is necessary to match the structure that Lambda expects. We can then zip up the ruby folder:

mv ruby/ruby ruby/gems
zip -r layer.zip ruby

Finally, publish the layer to AWS Lambda:

aws lambda publish-layer-version \
           --layer-name $LAYER_NAME \
           --region eu-west-1 \
           --compatible-runtimes ruby2.7 \
           --zip-file fileb://layer.zip

Make sure to set the region to the one you actually want to use the layer from. Every region has its own layers.

Note: If you want to deploy your layer to the public or use it more broadly across all regions, you'll want to search up the AWS Serverless Application Repository.