AWS SnapStart - Part 14 Measuring cold and warm starts with Java 21 using different compilation options

Introduction

In the previous parts we've done many measurements with AWS Lambda using Java 21 runtime with and without using AWS SnapStart and additionally using SnapStart and priming DynamoDB invocation :

• cold starts using different deployment artifact sizes
• cold starts and deployment time using different Lambda memory settings
• warm starts using different Lambda memory settings

We've done all those measurements using the following JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=1" defined in the AWS SAM template.yaml. This means that client compilation (c1) without profiling will be applied. It was considered the best choice due to the article Optimizing AWS Lambda function performance for Java by Mark Sailes. But all these measurements have been done for Java 11 and before Lambda SnapStart has been released. So now it's time to revisit this topic and measure cold and warm start times with different Java compilation options without SnapStart enabled, with SnapStart enabled (and additionally with priming). In this article we'll do it for Java 21 runtime and later in the subsequent parts with Java 17.

Meaning of Java compilation options

This picture shows Java compilation available.

If you don't specify any options, the default one applied for will be tiered compilation. You can read more about it in this article Tiered Compilation in JVM or generally about client (C1) and server (C2) compilation in the article Client, Server, and Tiered Compilation. There are also many other settings so you can apply to each of the compilation options. You can read more about them in this article JVM c1, c2 compiler thread – high CPU consumption?

Measuring cold starts and deployment time with Java 21 using different compilation options

For measurement purposes I created the sample application and configured Lambda functions to use Java 21 runtime for Lambda and 1024 MB memory which we introduced in part 9. Here is the code for our sample application

There are basically 2 Lambda functions which both respond to the API Gateway requests and retrieve product by id received from the API Gateway from DynamoDB. One Lambda function GetProductByIdWithPureJava21Lambda can be used with and without SnapStart and the second one GetProductByIdWithPureJava21LambdaAndPriming uses SnapStart and DynamoDB request invocation priming.

The results of the experiment below were based on reproducing more than 100 cold and approximately 100.000 warm starts. For it (and experiments from my previous article) I used the load test tool hey, but you can use whatever tool you want, like Serverless-artillery or Postman. I ran all these experiments with 5 different compilation options defined in the template.yaml. This happens in the Globals section where variable named "JAVA_TOOL_OPTIONS" is defined in the Environment section of the Lambda function:

Globals:
  Function:
    CodeUri: target/aws-pure-lambda-snap-start-21-1.0.0-SNAPSHOT.jar
    Runtime: java21
    ....
    Environment:
      Variables:
        JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=1"

1. no options (tiered compilation will take place)
2. JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=1" (client/C1 compilation without profiling)
3. JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=2" (client/C1 compilation with basic profiling)
4. JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=3" (client/C1 compilation with full profiling)
5. JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=4" (server/C2 compilation)

For their meaning see our explanations above. We will refer to those compilation options in the table column "Compilation Option" by their number in the tables below, for example number 5 stays for JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=4". Abbreviation c is for the cold start and w is for the warm start.

Cold (c) and warm (w) start times without SnapStart in ms:

Cold (c) and warm (w) start times with SnapStart without Priming in ms:

Cold (c) and warm (w) start times with SnapStart and with DynamoDB invocation Priming in ms:

Conclusions

For all measurements we discovered that setting compilation options -XX:+TieredCompilation -XX:TieredStopAtLevel= 2, 3 or 4 produced much worse cold and warm starts as the tiered compilation or -XX:TieredStopAtLevel=1 (client compilation without profling).

For the Lambda function without SnapStart enabled client compilation without profiling (-XX:+TieredCompilation -XX:TieredStopAtLevel=1 ) is the better option for having lower cold and warm starts for nearly all percentiles for our use case.

For the Lambda function with SnapStart enabled (with priming of DynamoDB invocation or without priming) tiered compilation (the default one) outperformed the client compilation without profiling (-XX:+TieredCompilation -XX:TieredStopAtLevel=1 ) in terms of the lower cold start time and also for the warm start time for nearly all percentiles for our use case.

So please review/re-measure the cold and warm start times for your use case if you use Java 21, as tiered compilation can be a better choice if you enable SnapStart for your Lambda function(s). In our case we didn't use any framework like Spring Boot, Micronaut or Quarkus which may also impact the measurements.

In one of my next articles I will make the same measurements but using Java 17 runtime.