Running Python 3.12 on AWS Lambda

AWS Lambda functions have completely revolutionized the way I work with (and think about) compute. It's just amazingly convenient to implement a quick function, push it to Lambda, schedule it with EventBridge and let it run for free or for almost free.

However, if you're like me, you want to keep pace with the latest and greatest in the Python world, which, at the time of this writing, is Python 3.12. I find the new error messages unbelievably useful and I could hardly wait for the ability to use the same quotation mark inside the curly braces in f-strings.

And, this passion to keep up with the latest and greatest interferes with using AWS Lambda, which only offers Python 3.7, 3., 3.9, 3.10 and 3.11.

Luckily, there's a way around this, and it's fairly easy to implement: create your own Lambda runtime.

There's ample documentation and a wide array of tutorials that show you how to do this (e.g. https://docs.aws.amazon.com/lambda/latest/dg/runtimes-walkthrough.html). However, I also like to follow the D.R.Y. (Don't Repeat Yourself) principle, so why not package it as a reusable Docker image and use it in all my Lambda apps?

Step 1. Setting things up

Let's start by creating a new directory for our entire build setup:

mkdir -p python-lambda-runtimes 
cd python-lambda-runtimes

Let's create a Dockerfile in the directory with the following content (we're going to look at what each line does after the code).

# Define custom function directory
ARG FUNCTION_DIR="/var/task/"

FROM python:3.12-slim-bookworm as build-image

# Include global arg in this stage of the build
ARG FUNCTION_DIR
RUN mkdir -p ${FUNCTION_DIR}

# Install aws-lambda-cpp build dependencies
RUN apt-get update && \
  apt-get install -y \
  g++ \
  make \
  cmake \
  unzip \
  libcurl4-openssl-dev

# Install the function's dependencies
RUN pip install --target ${FUNCTION_DIR} awslambdaric


FROM python:3.12-slim-bookworm

# Include global arg in this stage of the build
ARG FUNCTION_DIR
# Set working directory to function root directory
WORKDIR ${FUNCTION_DIR}

# Copy in the built dependencies
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]

Let's see what's happening here. We initiate a multi-phase build to reduce the size of our final image. On the FROM lines we choose which Python version we want to use for our runtime. If you wanted to work with Python 3.11 then you could simply replace the 3.12 part with 3.11.
As a last step of the first build phase we install awslambdaric, which is the AWS Lambda Runtime Interface Client which makes sure that the Lambda environment is able to communicate with our own code.
In the 2nd stage we simply copy the runtime interface client into our final image.

Step 2: Create a repository in your favorite Docker Container Registry

Since Docker has changed pricing model some time ago I started using AWS Elastic Container Registry (ECR) for my custom, private images. So, let's create a new repo. You can either use the AWS Console if you prefer the GUI, or if you like the CLI, you can simply run this command:

aws ecr create-repository --repository-name python-lambda-runtime

This will create a repository called python-lambda-runtimes. Feel free to replace the name with anything you prefer.

Step 3: Log in to your preferred repo

For example, if you are using ECR you can use this to log in:

aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin <AWS_ACCOUNT_ID>.dkr.ecr.eu-west-1.amazonaws.com

Please, replace with the ID of your AWS account.

Step 4: Build and push your image

To build your image and push it to ECR, you can do the following:

docker build -t python-lambda-runtime . && \
docker tag python-lambda-runtime:latest <AWS_ACCOUNT_ID>.dkr.ecr.eu-west-1.amazonaws.com/python-lambda-runtime:latest && \
docker push <AWS_ACCOUNT_ID>.dkr.ecr.eu-west-1.amazonaws.com/python-lambda-runtime:3.12

After this, you have everything in place to start using your pre-built image for your Lambda functions. In the next section, we'll do through all the steps you need to take to use it in your SAM template.

Step 5: Use the image in your SAM template

1. Create a Dockerfile in the directory of your Lambda function. Add the following content:

FROM <AWS_ACCOUNT_ID>.dkr.ecr.eu-west-1.amazonaws.com/python-lambda-runtime:3.12
COPY . /var/task/
RUN chmod -R 0755 .
RUN pip install -r requirements.txt
CMD ["app.lambda_handler"]

1. Look for the Type: AWS::Serverless::Function part of your template.yaml file. Remove the CodeURI, Handler, and the Runtime lines, and add PackageType: Image instead.
2. Add a new section with the same indentation as the Type: AWS::Serverless::Function with the following content:

Metadata:
    DockerTag: <NAME OF YOUR FUNCTION>
    DockerContext: ./<DIRECTORY OF YOUR FUNCTION>
    Dockerfile: Dockerfile

For example, if your function was called TestFunction and lived in the directory test_function of your project root, your Metadata would look like this:

Metadata:
    DockerTag: TestFunction
    DockerContext: ./test_function
    Dockerfile: Dockerfile

If you run sam build, it will create the new container image for you.

If you've already deployed your Lambda function before, you'll have to delete it or deploy the containerized one under a new name. Also, because your samconfig.toml file already contains settings for your previous deployment, it makes sense to rename it to something else, and run sam deploy --guided to re-populate it with your new settings.