DEV Community 👩‍💻👨‍💻

Cover image for How to create a Python Lambda Layer?
Gilbert Young Jr for AWS Community Builders

Posted on • Updated on

How to create a Python Lambda Layer?

In this article I intend to explain what a Lambda Layer is, the benefits of it, and lastly, how to create a Layer for a Python Lambda function.

What is a Lambda Layer?

A Lambda Layer is an isolated zip file that contains libraries, packages and/or application code that is shareable between your Lambda functions. A common use case for a Lambda Layer is, for example your company has two separate Python Web Scrapers. Both functions use pandas to do some sort of data processing. Since they both use the same Python packages, the smart solution is to create a Lambda Layer, so both functions can access the Layer. Below is a visual from AWS that shows a solution with Lambda Layers vs without Lambda Layers.


Benefits of Lambda Layers

  • Reduces the size of uploaded deployment archives.
  • Increases the speed of deployment.

Creating a Python Lambda Layer

We will create the layer using an EC2 Instance that will be provisioned by IaC (CloudFormation). I know what you are thinking. Why do we need an EC2 Instance to simply create a Lambda Layer? Why not just do it locally? The architecture for a Python Lambda Layer requires a Linux OS. Doing it on a Linux EC2 Instance helps us meet the following requirements:

  1. AWS Lambda runs on a Linux machine
  2. Any Python Packages added to the Layer need to be compiled with the correct architecture (Linux x86_64).

You can find the complete CloudFormation template on my GitHub.

Create stack parameters

    Type: String
    Description: Enter the name of your instance.
    Default: LinuxInstance

Enter fullscreen mode Exit fullscreen mode

The only parameter we will be using is for our instance name.

Create IAM Role & Instance Profile

# Instance Profile
    Type: AWS::IAM::InstanceProfile
      InstanceProfileName: !Join ["_", [!Ref InstanceName, "Profile"]]
      Path: /
        - !Ref InstanceRole

  # Instance Role
    Type: AWS::IAM::Role
      RoleName: !Join ["_", [!Ref InstanceName, "EC2PublishLambdaLayer"]]
        Version: 2012-10-17
          - Effect: Allow
              - sts:AssumeRole
      Path: /
        - PolicyName: !Join ["_", [!Ref InstanceName, "Policy"]]
            Version: "2012-10-17"
              - Effect: Allow
                Action: "lambda:PublishLayerVersion"
                Resource: "*"
Enter fullscreen mode Exit fullscreen mode

NOTE: The Policy Document contains the action lambda:PublishLayerVersion, without this policy we will not be able to publish the Lambda Layer from within the EC2 Instance.

Create EC2 Instance

 # EC2 Instance
    Type: AWS::EC2::Instance
      ImageId: ami-09d3b3274b6c5d4aa # Static AMI ID chosen from List of AMIs in the console
      InstanceType: t2.micro
      IamInstanceProfile: !Ref InstanceProfile
        - !Ref SSHSecurityGroup
      # Install Python, Pandas and Publish Lambda Layer
        Fn::Base64: |
          sudo amazon-linux-extras install python3.8
          curl -O
          python3.8 --user
          mkdir -p home/ec2-user/python
          cd home/ec2-user
          python3.8 -m pip install pandas -t python/
          zip -r python
          aws lambda publish-layer-version --layer-name pandas-layer --zip-file fileb:// --compatible-runtimes python3.8 --region us-east-1
        - Key: Name
          Value: !Ref InstanceName

Enter fullscreen mode Exit fullscreen mode

If you look at the UserData property, we are passing bash scripts. This is where the magic happens. We defined scripts to install Python and pandas. We then zip the directory that has the Python Package and lastly publish the Lambda Layer to AWS.

Upload CF Stack

To upload your CF stack, use the command below.

aws cloudformation deploy 
--template-file <path_to_file> 
--stack-name <stack_name> 
--region us-east-1 
Enter fullscreen mode Exit fullscreen mode

If the above command is successful, you will get an output in your terminal saying "Successfully created/updated stack - ". The stack should take a few minutes to provision our EC2 Instance and execute our scripts. Lastly, navigate to Lambda Layers in the AWS console and you should see the Layer that was created.

Lamda Layer

Clean Up

To clean up all the resources that were created simply delete the CloudFormation stack.

aws cloudformation delete-stack 
--stack-name <stack_name>
Enter fullscreen mode Exit fullscreen mode

I hope you found this article useful. Stay curious, keep learning, and keep building. Thank you.

Top comments (4)

avinashdalvi_ profile image
Avinash Dalvi

Good article. Small correction in title required. Lamda should Lambda.

giyoungjr profile image
Gilbert Young Jr

Appreciate it bro @avinashdalvi_ . Good looking out on the typo.

giyoungjr profile image
Gilbert Young Jr

Thank you @rishitharamesh1

18 Useful Github Repositories Every Developer Should Bookmark

>> Check out this classic DEV post <<