By default your AWS Lambda functions include only core Python functionality. Lambda layers are a fantastic feature allowing you to add external dependencies to your Lambda functions. In this post I'll walk you through how you can create your own library of Lambda layers using Docker.
What are Lambda Layers?
If you're unfamiliar with Lambda layers, a layer is additional code you can include at runtime with your Lambda functions to incorporate external libraries into your Lambda function. By default Lambda functions only include core Python libraries. Layers are a way for you to include other external libraries into your Lambda functions. A useful feature of layers is that they exist outside of the Lambda function itself. Once you create a layer you can use that layer across all your Lambda functions. You can read more about layers from the AWS documentation.
Creating a Lambda Layer with Docker
For this post we're going to create a layer using Flask as an example. Creating a Lambda Layer with Docket has four main steps:
- Setup the a local directory for the layer
- Use Docker to install the layer packages
- Create a zip file of your directory
- Upload to AWS using the AWS CLI
The next few sections will describe each of these steps in more detail. The last section will show how these steps can be combined into a single bash script to make deployment even easier.
Setting Up Your Layer Directory
The first step is to build the local directory structure that will contain all the artifacts for our layer. In the terminal enter the commands below. A new directory called flask-layer
is created along with a subdirectory python/lib.python3.8/site-packages
where we will install our Python packages to.
mkdir flask-layer
cd flask-layer
mkdir -pv python/lib/python3.8/site-packages
If you want to create your Layer for multiple Python versions you can repeat the last command and change "python3.8" to whatever other version of Python you want to include.
Within the flask-layer
directory create a requirements.txt
file that includes the following.
flask==1.1.1
The requirements.txt
file defines the packages that will be included in your layer. If you have additional dependencies you want to include in your layer you can add those to this requirements.txt
file.
One thing to be aware of with layers is that when they are loaded into your Lambda function they cannot exceed 250MB in size. A Lambda function can include up to 5 different layers. Given these limitations, having smaller layers you can mix-and-match across your functions is preferable rather than building large, single-purpose, layers.
Installing the Requirements with Docker
Docker will install the dependencies you specify in requirements.txt
. Docker has images to replicate the AWS Lambda environment. Using those images Docker will install the libraries you specified in requirements.txt
that are compatible with the Lambda runtime. In the terminal enter the following code. Make sure you have Docker running before entering this command!
docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"
After running the command you should see the site-packages
directory you created in the previous step populated with all your dependencies.
If you have additional versions you want to create your Layer for make sure you substitute the occurrences of "python3.8" with the correct version you want to use.
Uploading Your Layer to AWS
AWS requires all the layer code to be in a zip archive, so we need to zip everything in the python
directory.
zip -r flask-layer.zip python
All the contents of the python
directory are included in the zip archive called flask-layer.zip
.
Now the layer can be uploaded to AWS using the AWS CLI. You need to provide a few parameters in this step:
-
layer-name
is the name your want to give your layer -
description
to briefly summarize the layer -
zip-file
is the path to the zip archive you created in the previous step -
compatible-runtimes
details the Python versions your layer is compatible with
Enter the following into the terminal to publish our layer to AWS:
aws lambda publish-layer-version \
--layer-name "flask-layer" \
--description "Lambda Layer for Flask 1.1.1" \
--zip-file "fileb://flask-layer.zip" \
--compatible-runtimes "python3.8"
And with that your Flask layer is published to AWS! Now let's simplify this process a little more by packaging some of these steps into a single bash script.
Creating Your Own Lambda Layer Directory
To manage my layers I created a single bash script packaging most of the steps above together to make deploying layers easier. I have a directory where I manage all my layers. Below is a sample of those layers. Each layer has its own directory and the create-lambda-layer.sh
script handles installing the layer dependencies and uploading them to AWS.
Here is what create-lambda-layer.sh
looks like:
#!/usr/bin/env bash
while [[ "$#" -gt 0 ]]; do case $1 in
-v|--version) pythonEnvs+=("$2"); shift;;
-n|--layer-name) layerName="$2"; shift;;
-d|--desc) layerDescription="$2"; shift;;
*) echo "Unknown parameter passed: $1"; exit 1;;
esac; shift; done
# Create and install requirements to directory.
for penv in ${pythonEnvs[@]}; do
mkdir -pv python/lib/${penv}/site-packages
docker run -v "$PWD":/var/task "lambci/lambda:build-${penv}" /bin/sh -c "pip install -r requirements.txt -t python/lib/${penv}/site-packages/; exit"
done
# Create zip file of environments.
zip -r ${layerName}.zip python
# Publish Layer to Lambda.
aws lambda publish-layer-version \
--layer-name "${layerName}" \
--description "${layerDescription}" \
--zip-file "fileb://${layerName}.zip" \
--compatible-runtimes ${pythonEnvs[@]}
The script accepts parameters for Python versions (you can provide multiple!), a name and description for your layer. From those parameters the script will generate the file structure, run the docker command for each Python version, create the zip archive, and then update the layer to AWS.
We can test this by installing the Twilio helper library. In the Terminal, create a directory where you're going to store your Lambda layers.
mkdir my-lambda-layers
cd my-lambda-layers
Create a file inside that directory, create-lambda-layer.sh
, and paste the code above into the file.
Each layer needs to have its own directory, run the following code in the Terminal to create a directory twilio-layer
.
mkdir twilio-layer
cd twilio-layer
The last bit of setup is to create the requirements.txt
file, which should include:
twilio==6.40.0
Now we can call our bash script to finish creating and deploying our layer (again, make sure you have Docker running before executing this code!)
bash ../create-lambda-layer.sh -v "python3.8" -n "twilio-layer" -d "Twilio 6.40.0 Layer"
And your layer is deployed to AWS!
If you want to create your layer for multiple different Python versions all you have to do is specify additional -v
parameters with those versions. The example below creates a layer for the Python 3.6, 3.7 and 3.8 runtimes.
bash ../create-lambda-layer.sh -v "python3.8" -v "python3.7" -v "python3.6" -n "twilio-layer" -d "Twilio 6.40.0 Layer"
Easy Peasy!
Wrapping Up
In this post I showed you how to deploy a Lambda Layer through the commandline using Docker. The bash script provided isn't fully featured, but in its current form provides a convenient way deploy your Lambda layers. Layers are a neat tool that can help you add functionality to your Lambdas. Docker simplifies creating layers so you can manage your own library of layers that you can share with your friends and coworkers!
Happy Building!
Top comments (1)
Great documentation. Thank you 😊