Custom AWS Lambda Docker image for local development

tl:dr; Custom docker image for AWS Lambda Functions that support local TypeScript development, hot reloading and testing

The Dilemma

I recently had to make a serverless function, that converts files from one format to another. Before writing a single line of code I laid some ground rules for the project:

• It needs to run in AWS Lambda
• Use CLI tools to convert the files
• Development must be smooth and local
• Local environment must be as close to the real deal as possible

Now there are few problems with this set of rules:

• The default Lambda Node.js runtime doesn't have CLI access or the CLI tools needed
• Local Lambda development is extremely cumbersome and hard

Luckily there is one solution for both of these problems: Lambda Container Images!

The Idea and Theory

When starting on this journey I quite quickly realized there is not much information on projects like this and the use case is super niche.

To run something in AWS Lambda we need two basic things. Lambda Runtime Interface Client (aws-lambda-ric) and Lambda Runtime Interface Emulator (aws-lambda-rie)

For the production image we only need to provide Runtime + Function but for the development image we also need to provide the Runtime API

AWS also generously provides base images for multiple languages and even vanilla ones that have no runtime by default. Sadly for us, these all use the CentOS like Amazon Linux and do not offer hot reloading.

If you are fine with using EPEL packages and don't need hot reloading I suggest you use the provided base images and turn away now.

The Execution part 1: Getting started

Oh you are still here? Welcome aboard for the ride!

In my use case I needed some very specific Debian packages and the project would be complex enough, that hot reloading in local development would be nice. So custom Docker image it is!

I opted to craft two different images. One for local development and testing, the other for production. The local one will use only one stage but for the production one we will use a multi-stage Dockerfile to slim down the final image and make lambda start-ups as fast as possible.

We will also have a docker-compose.yml to make local running convenient. The compose will spin up 2 versions of the development image. One which will run the tests and one that will run the dev server.

So go ahead and initialize the project any way you would like. I will use file structure like so with yarn as the package manager:

├── function
│   ├── index.ts
|   tests
│   ├── index.spec.ts
│   package.json
│   tsconfig.json
│   Dockerfile
│   docker-compose.yml

During this deep dive I will also use esbuild and Nodemon. These will be used for watching, bundling and transpiling the TypeScript code but you can replace them with whatever you want.

For testing I will use Mocha with Chai assertions but you are once again free to use what ever you want. I will also use ts-node to run Mocha inside the container.

Some other useful dependencies might include:

• @types/aws-lambda
• @types/node

For the complete setup go and have a peek here as I wont be explaining it all in this post.

You will also need to download the aws-lambda-rie binary so that we can copy it to the development image.

The Execution part 2: Code

To see if the images work we need something to run in them. So go ahead and create a function/index.ts file with the following code:

import { APIGatewayEvent, APIGatewayProxyResult, Context } from 'aws-lambda'

export const handler = async (event: APIGatewayEvent, context: Context): Promise<APIGatewayProxyResult> => {
  console.info('EVENT\n' + JSON.stringify(event, null, 2))

  // The local aws-lambda-rie will pass data straight to the event but the real one to event.body
  const data = JSON.parse(event.body || JSON.stringify(event))
  console.info('DATA\n' + JSON.stringify(data, null, 2))

  return {
    statusCode: 200,
    body: JSON.stringify({
      message: 'Hello from Lambda!',
    }),
  }
}

In here you can do anything you would normally do in Lambda functions but today we will KISS and just use hello-world!

One notable thing is that for some reason the runtime emulator AWS provides passes POST data straight to the event object while the real runtime in cloud puts it to event.body. This might pose some problems if you use pre-made adapters e.g. aws -> express.

An example of a code used for testing could be (tests/index.spec.ts):

import chai from 'chai'
import chaiHttp from 'chai-http'
import { describe } from 'mocha'

chai.use(chaiHttp)
chai.should()

const baseUrl = 'http://localhost:8080/2015-03-31/functions/function/invocations'

describe('it-runs', () => {
  it('Should return 200', () => {
    chai
      .request(baseUrl)
      .get('/')
      .end((err, res) => {
        res.should.have.status(200)
      })
  })
})

The Execution Part 3: Main Course

When building Docker images I like to start with the bigger ones and then optimize away as I go towards a production version, so lets start with the Development image.

It needs to atleast:

• Have the Lambda runtime for Node.js
• Have the Lambda runtime interface emulator
• Update the code inside the container when local code changes
• Bundle the TypeScript code to JavaScript for execution on changes
• Restart the RIE and RIC after code changes

So lets get started (Final file):

First things first let's initialize the Docker image. For this I will use node:18-bullseye image

FROM node:18-bullseye
WORKDIR /usr/app

After that we have to install all the cpp dependencies needed to build the aws-lambda-ric for Node.js. This is a really slow step so it's a good idea to keep it as high in the Dockerfile as possible to maximize the re-usage of cached layers.

RUN apt-get update && \
    apt-get install -y \
    g++ \
    make \
    cmake \
    unzip \
    libcurl4-openssl-dev \
    lsof

This is also a good place to install any dependencies you might need e.g.:

RUN apt-get install -y \
  inkscape \
  imagemagick

Copy only the package.json and other necessary files to install dependencies for the same cached layers reason:

COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile

Now we are ready to install the aws-lambda-ric

RUN yarn add aws-lambda-ric

After this we can copy the rest of stuff over. Just make sure you have a .dockerignore to exclude node_modules and other unwanted stuff.

COPY . .

To bundle and transpile the code we can use the following esbuild commands (package.json):

"scripts": {
  "build:dev": "esbuild function/index.ts --platform=node --bundle --target=node14 --outfile=dist/index.js",
  "build": "esbuild function/index.ts --platform=node --bundle --minify --target=node14 --outfile=dist/index.js"
}

Redoing the build on code changes is quite easy with Nodemon but what's not easy is to restart the RIC and RIE. They are designed to be run in production enviroments and are harder to kill than cockroaches. They also exit with the code 2 which is no bueno for Nodemon.

To make all this bit simpler I devised a entrypoint.dev.sh which is just a bash script that kills everything running in port 8080 and then rebuilds the code and restarts the RIC and RIE processes.

#!/bin/bash

PID=$(lsof -t -i:8080)
if [ -z "$PID" ]
then
    echo "No PID found"
else
    echo "Killing PID $PID"
    kill $PID
fi

yarn build:dev && ./aws-lambda-rie yarn aws-lambda-ric dist/index.handler || exit 1

It also makes sure to always exit with code 1 to keep Nodemon happy. Now we can just create a nodemon.json to run this:

{
  "watch": ["function"],
  "ext": "ts,json",
  "exec": "./entrypoint.dev.sh"
}

and copy all the necessary files and add their execution rights:

COPY entrypoint.dev.sh ./
RUN chmod +x ./entrypoint.dev.sh

COPY aws-lambda-rie ./aws-lambda-rie
RUN chmod +x ./aws-lambda-rie

Toss in a default start command for good measure and we are almost ready to go:

CMD ["yarn", "nodemon"]

Now the only thing left to do is figuring out how to pass the code changes from local files to the container. Luckily this is quite easy and we can just mount the our local code folder as a volume in the container.

To make this process and running the container in general easier I made the following docker-compose.yml:

services:
  development:
    build:
      context: .
      dockerfile: Development.Dockerfile
    ports:
      - 9000:8080
    volumes:
      - ./function:/usr/app/function

So now if you just run:

docker compose  -f "docker-compose.yml" up -d --build development

Followed by:

curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"body": "test"}'

You should get Hello World back! Go ahead and try editing and saving the code. You should see a updated response!

The Execution Part 4: Testing

For testing we don't need no Nodemon and hot reloading but instead we have to run the testing library, while the RIE&RIC combo is running, and then exit the container image with the exit code of the testing library. This way we can use the testing image in e.g. pipeline quite handily.

To get started let's make a new entrypoint file entrypoint.test.sh:

#!/bin/bash

PID=$(lsof -t -i:8080)
if [ -z "$PID" ]
then
    echo "No PID found"
else
    echo "Killing PID $PID"
    kill $PID
fi

yarn build
nohup ./aws-lambda-rie yarn aws-lambda-ric dist/index.handler > /dev/null 2>&1 &
yarn mocha

As was with the development version, we first kill everything in the 8080 port, then we build the code using the production configuration (Difference to development being the --minify flag). After this we need to spin up the Lambda Runtimes, forget about them and move on. This can be achieved with the nohup <command> & I will also pipe the output of these to /dev/null so they don't bother us.

As the last command you can run your testing library of choice any way you want. For me this is with the yarn mocha command in combination with the .mocharc.json file:

{
  "extension": ["ts"],
  "spec": "tests/**/*.spec.ts",
  "require": "ts-node/register"
}

Don't forgot to add these to the Development.Dockerfile above the CMD:

COPY entrypoint.test.sh ./
RUN chmod +x ./entrypoint.test.sh

To make things easier we can also add this to the docker-compose.yml:

tests:
  build:
    context: .
    dockerfile: Development.Dockerfile
  command: ./entrypoint.test.sh
  environment:
    - NODE_ENV=test

The important part here is how we override the default CMD with our own entrypoint.

The Execution Part 5: Production

To make a production ready image we just need slight modifications to our Development.Dockerfile so go ahead and create a Dockerfile and put in it:

FROM node:18-bullseye as build-image

WORKDIR /usr/app

# Install aws-lambda-ric cpp dependencies
RUN apt-get update && \
    apt-get install -y \
    g++ \
    make \
    cmake \
    unzip \
    libcurl4-openssl-dev \
    lsof


# Install dependencies
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
RUN yarn add aws-lambda-ric

# Copy the source code
COPY function ./function
COPY tsconfig.json ./

# Build the minified and bundled code to /dist
RUN yarn build

# Create the final image
FROM node:18-bullseye-slim

WORKDIR /usr/app

# Copy the built code from the build image
COPY --from=build-image /usr/app/dist ./dist
COPY --from=build-image /usr/app/package.json ./package.json

RUN apt-get update

# Install any dependencies you might need e.g.
# RUN apt-get update && apt-get install -y \
#   inkscape \
#   imagemagick

# Install the aws-lambda-ric
COPY --from=build-image /usr/app/node_modules/aws-lambda-ric ./node_modules/aws-lambda-ric

# Run the aws-lambda-ric
ENTRYPOINT [ "node", "node_modules/aws-lambda-ric/bin/index.js" ]
CMD [ "dist/index.handler" ]

The process is almost identical but we do not need the aws-lambda-rie as it will be provided by the AWS Servers. We also use 2 stage build and use the node:18-bullseye-slim for the final image. This allows us to leave the node_modules and other unnecessary clutter in the first stage.

In my testing we go from a >1Gb Development image to <350Mb Production image which is quite the saving.

To publish and use this image in Lambda we can e.g. use AWS-CDK deployment pipeline or push it to a container registry in a custom pipeline and download it from there to Lambda.

If you would like to see a tutorial on that please leave a like and comment 🙌🏻