In this article I intend to explain what a Lambda Layer is, the benefits of it, and lastly, how to create a Layer for a Python Lambda function.
What is a Lambda Layer?
A Lambda Layer is an isolated zip file that contains libraries, packages and/or application code that is shareable between your Lambda functions. A common use case for a Lambda Layer is, for example your company has two separate Python Web Scrapers. Both functions use pandas to do some sort of data processing. Since they both use the same Python packages, the smart solution is to create a Lambda Layer, so both functions can access the Layer. Below is a visual from AWS that shows a solution with Lambda Layers vs without Lambda Layers.
Benefits of Lambda Layers
- Reduces the size of uploaded deployment archives.
- Increases the speed of deployment.
Creating a Python Lambda Layer
We will create the layer using an EC2 Instance that will be provisioned by IaC (CloudFormation). I know what you are thinking. Why do we need an EC2 Instance to simply create a Lambda Layer? Why not just do it locally? The architecture for a Python Lambda Layer requires a Linux OS. Doing it on a Linux EC2 Instance helps us meet the following requirements:
- AWS Lambda runs on a Linux machine
- Any Python Packages added to the Layer need to be compiled with the correct architecture (Linux x86_64).
You can find the complete CloudFormation template on my GitHub.
Create stack parameters
---
Parameters:
InstanceName:
Type: String
Description: Enter the name of your instance.
Default: LinuxInstance
The only parameter we will be using is for our instance name.
Create IAM Role & Instance Profile
# Instance Profile
InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
InstanceProfileName: !Join ["_", [!Ref InstanceName, "Profile"]]
Path: /
Roles:
- !Ref InstanceRole
# Instance Role
InstanceRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ["_", [!Ref InstanceName, "EC2PublishLambdaLayer"]]
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
Path: /
Policies:
- PolicyName: !Join ["_", [!Ref InstanceName, "Policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: "lambda:PublishLayerVersion"
Resource: "*"
NOTE: The Policy Document contains the action lambda:PublishLayerVersion, without this policy we will not be able to publish the Lambda Layer from within the EC2 Instance.
Create EC2 Instance
# EC2 Instance
Instance:
Type: AWS::EC2::Instance
Properties:
ImageId: ami-09d3b3274b6c5d4aa # Static AMI ID chosen from List of AMIs in the console
InstanceType: t2.micro
IamInstanceProfile: !Ref InstanceProfile
SecurityGroups:
- !Ref SSHSecurityGroup
# Install Python, Pandas and Publish Lambda Layer
UserData:
Fn::Base64: |
#!/bin/bash
sudo amazon-linux-extras install python3.8
curl -O https://bootstrap.pypa.io/get-pip.py
python3.8 get-pip.py --user
mkdir -p home/ec2-user/python
cd home/ec2-user
python3.8 -m pip install pandas -t python/
zip -r layer.zip python
aws lambda publish-layer-version --layer-name pandas-layer --zip-file fileb://layer.zip --compatible-runtimes python3.8 --region us-east-1
Tags:
- Key: Name
Value: !Ref InstanceName
If you look at the UserData property, we are passing bash scripts. This is where the magic happens. We defined scripts to install Python and pandas. We then zip the directory that has the Python Package and lastly publish the Lambda Layer to AWS.
Upload CF Stack
To upload your CF stack, use the command below.
aws cloudformation deploy
--template-file <path_to_file>
--stack-name <stack_name>
--region us-east-1
--capabilities CAPABILITY_NAMED_IAM
If the above command is successful, you will get an output in your terminal saying "Successfully created/updated stack - ". The stack should take a few minutes to provision our EC2 Instance and execute our scripts. Lastly, navigate to Lambda Layers in the AWS console and you should see the Layer that was created.
Clean Up
To clean up all the resources that were created simply delete the CloudFormation stack.
aws cloudformation delete-stack
--stack-name <stack_name>
I hope you found this article useful. Stay curious, keep learning, and keep building. Thank you.
Top comments (3)
Good article. Small correction in title required. Lamda should Lambda.
Appreciate it bro @avinashdalvi_ . Good looking out on the typo.
Thank you @rishitharamesh1