DEV Community

loading...

Using external Python packages with AWS Lambda layers

mmascioni profile image Matthew Mascioni ・6 min read

Amazon Lambda is an awesome service that lets you run code serverlessly: rather than keeping a server running to execute your code anytime, a server is spun up only when necessary in response to some event. This event could be from an HTTP request, a message posted to a queueing service like Amazon SQS, or even when a file is uploaded to an S3 bucket!

Lambda supplies different runtimes depending on what language you want to write your function in. For Python programming, this runtime includes the Python standard library, but what if you want to use external packages from PyPI or elsewhere? This is something you can do with Lambda layers! Layers provide dependencies (or even a custom runtime) for your function, and in this quick tutorial we'll walk through how to use them!

What you'll need

  • An AWS account and a little experience using Lambda functions. If you haven't used them before, Amazon has great tutorials!
  • Docker installed on your computer, as we'll use that to install dependencies in a similar environment to the AWS Lambda Python runtime

Our demo project will be a very simple Lambda function that accesses the PokéAPI's /api/v2/pokemon endpoint to return information about a Pokémon, given a name passed in the event that triggers the function.

Setting up our project locally

First we'll create a virtual environment for our project and work in there, to keep any project dependencies separate from others on your computer:

$ mkdir poke-lambda 
$ cd poke-lambda
$ python -m venv venv
$ source venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

We're using the requests library to do our API calls, so install that with pip install requests. Our one file, lambda_function.py looks like this:

import json
import requests

def lambda_handler(event, context):
    # Make sure a Pokemon name (or ID) was passed in the event object:
    pokemon = event.get('pokemon')
    if pokemon is None:
        return {
            'statusCode': 400,
            'body': json.dumps('Missing pokemon attribute in the event object')
        }

    # If we have a pokemon name/ID passed in, try to get info for it:
    r = requests.get(f"https://pokeapi.co/api/v2/pokemon/{pokemon}")

    if r.status_code == 200:
        status_code = 200
        body = json.dumps(r.json())
    else:
        status_code = r.status_code
        body = json.dumps(f"Could not load information for Pokemon {pokemon}")

    # Return what we got from the API:
    return {
        'statusCode': status_code,
        'body': body
    }
Enter fullscreen mode Exit fullscreen mode

What this code does is check for the pokemon attribute in the event data the Lambda function receives, queries the PokéAPI for information about the Pokémon passed in, and returns the API's response back (with basic error handling). At this point our project structure looks like this:

poke-lambda
|   lambda_function.py
└───venv
Enter fullscreen mode Exit fullscreen mode

Building our Lambda layer

In order to put this code on AWS Lambda, we need a way to include the requests library first! Lambda layers are .zip files containing libraries (and a custom runtime if you need) your function requires. For Python lambda layers, AWS requires libraries to be placed in either the python or python/lib/python-3.x/site-packages folders.

  1. While the virtual environment is activated for your project, save a list of its dependencies to a requirements.txt file: pip freeze > requirements.txt.

  2. Create a python folder. This will house all the libraries to go in our layer. At this point, your folder structure should be:

    poke-lambda
    | lambda_function.py
    | requirements.txt
    └───venv
    └───python
    
  3. From the project root, run the following command to build your dependencies in a container similar to the Lambda execution environment Amazon provides (courtesy of lambci):

    $ docker run --rm \
    --volume=$(pwd):/lambda-build \
    -w=/lambda-build \
    lambci/lambda:build-python3.8 \
    pip install -r requirements.txt --target python
    

    Let's break down what this command does, argument-by-argument:

    • --rm: Makes sure that once the Docker container exits, it's removed (keeps things clean)
    • --volume: Bind mounts the current working directory to the /lambda-build directory on the container (this allows the container to access the requirements.txt file we've just generated, and add files to the python directory). If you're on Windows, you can also paste in a full path to your project root instead of $(pwd).
    • -w: Sets the working directory in the container (in this case, to our project root)
    • lambci/lambda:build-python3.8: The docker image to run this container from -- make sure it matches the Python version you're working with! For reference, Amazon currently provides runtimes for Python 2.7, 3.6, 3.7 and 3.8 (I'm using 3.8 here)
    • pip install -r requirements.txt --target python: Installs the project dependencies as normal, from the requirements.txt file to the python folder (more on that here)
  4. You should see some output from pip. This is running in the Docker container. If successful, you should have dependencies installed to the python folder you created earlier! Mine looks like this:

    poke-lambda
    | lambda_function.py
    | requirements.txt
    └───venv
    └───python
        └───bin
        └───certifi
        └───certifi-2020.11.8.dist-info
        └───chardet
        └───chardet-3.0.4.dist-info
        └───idna
        └───...
    
  5. At this point, all we have to do is zip our python folder: zip -r layer python/. This will create a layer.zip file in your project's root directory.

  6. Next, upload the .zip file to Lambda! Sign into your AWS Console and head over to Services > Lambda > Layers (it should be under "Additional resources"). Click "Create layer" and give your layer a name! Pick the runtime that corresponds to the version of Python you're trying to use in your function, and upload the .zip file you created earlier:

Filling out the name, runtime and layer .zip file in the Create Layer form

Adding the layer to our Lambda function

We'll walk through this using the front-end console, but you can totally do this all over the AWS CLI!

  1. While in your AWS Lambda console, click "Create function". Set it up however you want (pick a Python runtime though)

  2. In the Lambda function designer, click on the "Layers" button:

    Clicking on the Layers button in the designer

  3. A table full of layers your function is using should appear below the designer. There aren't any so far, so lets add ours! Click "Add a layer" and choose "Custom layers". You should be able to see your layer you created earlier in the dropdown. Select and add it!

  4. Once added, you'll be taken to the Designer again. Scroll down and paste your function code in (you can alternatively upload a .zip file containing just the lambda_function.py file since that's all we need here)

    Pasting in our function code from before

  5. Save and click "Deploy" to update your function's code. Let's send our code a test event! At the top of the page, click on "Configure test events" under "Select a test event". We'll create an event payload to fetch details about Charizard:

    Configuring the test event for the Lambda function. This is a JSON payload with only one key (pokemon) equal to Charizard

  6. Save your test event, pick it from the dropdown and click "Test". You can now see your function returning results!

    Testing the Lambda function with the test event configured in step 5 should result in a success

That's all there is to it! Keep in mind functions can have multiple Lambda layers (up to 5 right now) and functions can share layers too.

In this tutorial we:

  • Wrote a simple Lambda function that calls an external API and returns the results
  • Created a Lambda layer containing all dependencies for our project that we can reuse in other projects later, using the lambci/lambda images and the Lambda GUI

Keep in mind all of this can also be accomplished over the AWS CLI (uploading the function, creating the layer and so on) if you prefer this method. There are lots of AWS docs which document this pretty well!

Hope you enjoyed this tutorial! Let me know if it was helpful in your project 😄

Discussion (6)

pic
Editor guide
Collapse
uraniumreza profile image
Nayeem Reza

Hey Matthew, I followed your instruction but somehow it's not working. I'm using a package named haversine, when I hit the API Gateway by using curl, I got this error:

{"errorMessage": "Unable to import module 'lambda_function': No module named 'haversine'", "errorType": "Runtime.ImportModuleError"}
Enter fullscreen mode Exit fullscreen mode

Any idea why it's happening? Or, how to solve this?

Collapse
mmascioni profile image
Matthew Mascioni Author

Hi Nayeem! Looks like the layer dependencies might not be available in your function. A few places this could've gone wrong:

  • Were there any errors raised while you were building dependencies using the Docker image?
  • Does the runtime Python version on your Lambda layer match the one of your Lambda function, and were there any errors raised when adding the layer to your function?
Collapse
uraniumreza profile image
Nayeem Reza

No, I haven't faced any error while building the Docker Image, also no errors while adding the layer to the function! And I'm using Python3.7 in both the function and layer runtime!

Thread Thread
mmascioni profile image
Matthew Mascioni Author

Super weird! Sorry it's been causing problems - are you able to share the code you're using for your function? I gave it a shot using the haversine package as well, hard-coded one of their examples in and it seemed to work - attached a gist of the function here (didn't use it with API Gateway though)

Collapse
stevematdavies profile image
Stephen Matthew Davies

Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

Collapse
mmascioni profile image
Matthew Mascioni Author

Hey Stephen! What step are you getting this error at?