Using Native Python Libraries in Lambda

It is possible to use native libraries in Amazon Lambda. In this example, we will be building a native bcrypt library against libcrypt, using virtualenv to manage our libraries.

Create an Amazon EC2 Instance to build the native library

Amazon Lambda runs on the Amazon Linux distribution, so we will need to target our compiled libraries against this distribution. Unfortunately, there doesn't appear to be any sort of debootstrap equivalent for Amazon Linux, so an instance using Amazon Linux will be needed for the build process.

1. Create an instance using Amazon Linux

2. Connect to the server

Set up the build environment on the server

We will need to install the build tools for the library we are building, as well as any build dependencies that are required.

$ sudo yum update
$ sudo yum install -y gcc44 gcc-c++ libgcc44 cmake
$ sudo yum install -y python27-devel python27-pip gcc libjpeg-devel zlib-devel gcc-c++
$ sudo yum install libffi-devel

Update/upgrade pip and setuptools

Make sure you are using the latest version of pip to prevent any issues.

$ pip update
$ sudo pip install --upgrade pip setuptools

Set up Virtualenv for your project

We will need to package any libraries that the lambda requires along with the lambda. Virtualenv provides us with a convenient method to bundle the libraries along with our lambda function.

Create a directory for your project, and enter it

$ mkdir $WORK
$ cd $WORK
$ virtualenv venv

Activate virtualenv and install libraries

Here we will build and install the libraries into the virtualenv directory, which will make them available for packaging later.

$ source venv/bin/activate
$ pip install --upgrade pip
$ pip install bcrypt

Place your lambda function into the working directory.

$ cp ~/lambda_function.py $WORK

Verify the script is working as expected.

Run your lambda and/or tests locally to verify all the libraries are working as expected.

Package the project, according to the lambda requirements.

Here's the tricky part -- Amazon Lambda requires all the libraries to be at the top level in the zip file that you upload. The following will create a zip file with the lambda function and all required libraries at the top level of the zip file. Since virtualenv contains all the libraries that we need for the project, we will copy all the libraries from the virtualenv environment to the zip file.

$ zip -9 bcrypt_lambda.zip lambda_function.py
$ cd $VIRTUAL_ENV/lib/python2.7/site-packages
$ zip -r9 $WORK/bcrypt_lambda.zip *
$ cd $VIRTUAL_ENV/lib64/python2.7/site-packages
$ zip -r9 $WORK/bcrypt_lambda.zip *

Upload the zip to Amazon Lambda.

Now that you have the zip file, upload it to Amazon Lambda as you normally would. Remember to set up the test data for the lambda to verify that the function is working as expected.