Benchmarking Gradle Builds Using Gradle-Profiler

In this post, we’re going to take an introductory look at benchmarking Gradle build performance using the gradle-profiler tool.

By the end, you should have a basic understanding of how to use gradle-profiler to gain better insights into the performance of your Gradle build and how you might use those insights to improve Gradle build performance for your project.

What is the Gradle-Profiler Tool?

The gradle-profiler project describes itself in the following way:

“A tool to automate the gathering of profiling and benchmarking information for Gradle builds.”

What does that mean to you and your project?

Imagine you want to get a sense of how fast your Gradle build is.

You might run your target Gradle task, wait for the build to complete, and take that build time as your result.

Now, because build times are often variable, you may want to run your Gradle task multiple times and average the execution times. So, you kick off your build, wait for it to finish, write down the total execution time, and repeat the process.

This process of manually benchmarking your Gradle build will likely take a while. Depending on how long your build takes to complete, and how many interactions you want to run, you could find yourself repeatedly coming back to your computer to check whether it’s time to start the next build or not. Even if you stay busy with other things, this is still a drawn out, tedious process that relies on you, as the developer, manually recording build statistics for each iteration.

It’s this process that gradle-profiler aims to automate for you.

Rather than you having to manually kick off each build and recording the resulting build statistics, gradle-profiler allows devs to start a single command which will repeatedly run a specified Gradle task and record all the build stats in an easy to examine output format.

This takes Gradle build benchmarking from a tedious, manual task to something that can be highly automated with little need for human interaction or monitoring.

I’ve been using gradle-profiler recently in an effort to keep my primary project’s build times low, and to examine the build impact of proposed changes. In this post, we’re going to walk through how you can start using gradle-profiler to gather benchmark data for your Gradle build, and how you may start using that data to understand the impact of changes on your project.

Installing Gradle-Profiler

Before you can start benchmarking Gradle build performance, you’ll need to install gradle-profiler to your development machine.

You can do this in one of several ways

Install From Source

You could clone the git repository to your local machine, and build the project from source.

→ git clone git@github.com:gradle/gradle-profiler.git
→ cd gradle-profiler
→ ./gradlew installDist

You would likely then want to add gradle-profiler/build/install/gradle-profiler/bin to your path or create some kind of alias that enables you to invoke the tool by executing the gradle-profiler command.

Install With Homebrew

If you are using Homebrew on your machine, installation is quite simple.

→ brew install gradle-profiler

Other Installation Options

If building from source, or installing using Homebrew, aren’t good options for you, you could try either of the following installation methods:

• Installing using SDKMAN
• Downloading binaries directly from GitHub

Examples On GitHub

You can find example commands and example benchmark scenarios in my sample repo on GitHub.

Find The Code

Benchmarking Your Gradle Build

Now that gradle-profiler is installed, and the gradle-profiler command is available to us, let’s start benchmarking Gradle build performance for your project.

Running the Benchmarking Tool

To generate benchmarking results for our project, we need two things:

1. The path to the project directory containing the Gradle project
2. A Gradle task to run

With these, we can start benchmarking our build like this:

→ cd <project directory>→ gradle-profiler --benchmark --project-dir . assemble

Let’s break down this command into its individual parts.

• gradle-profiler – invokes the gradle-profiler tool
• --benchmark – indicates that we want to benchmark our build
• --project-dir . – indicates that the Gradle project is located within the current working directory
• assemble – this is the Gradle task to benchmark

When this command is run, the benchmarking tool will begin. You should see output in your console indicating that warm-up and measured builds are running.

When all the builds are completed, two benchmarking artifacts should be created for you:

1. benchmark.csv – provides benchmarking data in a simple .csv format
2. benchmark.html – provides an interactive webpage report based on the .csv data

The output paths should look something like this:

Results written to /Users/n8ebel/Projects/GradleProfilerSandbox/profile-out
/Users/n8ebel/Projects/GradleProfilerSandbox/profile-out/benchmark.csv
/Users/n8ebel/Projects/GradleProfilerSandbox/profile-out/benchmark.html

Understanding Benchmarking Results

Once your outputs are generated, you can use them to explore the benchmarking results.

Interpreting .csv results

Here’s a sample benchmark.csv output generated by benchmarking the assemble task for a new Android Studio project.

This report is very streamlined and highlights only a few things. The most interesting data points are likely:

• Which version of Gradle was used
• Which Gradle tasks were run
• The task execution time, in milliseconds, for the warm-up and measured builds

Notice that the first warm-up build took significantly longer than every other build? Is this a problem? Is something wrong with your project’s configuration?

By default, gradle-profiler uses a warm Gradle daemon when measuring build times. If you’re not familiar, the Gradle daemon runs in the background to avoid repeatedly paying JVM startup costs. It can drastically improve the performance of your Gradle builds.

So, for this first warm-up build, task execution time is much longer as the daemon is started. After that, you can see that subsequent builds are much faster.

If we ignore the warm-up builds, and look only at the set of measured builds, we see that the build times are consistently fast as one might expect for a new project.

Interpreting HTML results

In addition to being an interactive webpage, the benchmarking.html results provide more data than the .csv file. By viewing the results this way, you’ll have access to:

• Mean, Median, StdDev, and other statistics from the measured build results
• Gradle argument details
• JVM argument details
• And more…

The HTML results provide a graph of each warm-up and measured build to help you visually understand performance over time.

If you aren’t interested in automating the analysis of your benchmarking results, then viewing the HTML results is often the most convenient way to quickly understand how long your build takes, and to quickly see if there are any outlier executions to examine.

This HTML view becomes even more useful when benchmarking multiple scenarios at the same time as we’ll see later on.

Detecting Gradle Performance Issues

Now that we have some understanding of the output generated by gradle-profiler benchmarking, let’s explore how we might begin to use that benchmark data to compare the performance of our Gradle builds.

The simplest approach is generally as follows:

• Collect benchmarking data for your current/default Gradle configuration
• Change your Gradle configuration
• Collect benchmarking data for updated configuration
• Compare the results

We could use this approach to compare the impact of caching on a build. We might compare the performance of clean builds versus incremental builds.

Any tweak to our build settings or project structure could be examined through this type of comparison.

Comparing clean and up-to-date build performance

Let’s walk through a quick example of this kind of analysis using gradle-profiler. We’re going to compare the impact of an up-to-date build over a clean build.

First we’ll benchmark up-to-date builds of our assemble task and collect the results.

→ gradle-profiler --benchmark --project-dir . assemble

We’re referring this as the up-to-date scenario because after the first execution of the assmble task, each task should be up-to-date and subsequent builds should be extremely fast as there is nothing to rebuild.

Next, we’ll benchmark our clean build.

→ gradle-profiler --benchmark --project-dir . clean assemble

In the clean build scenario, we are discarded previous outputs before executing our assemble task, so we should expect the clean build to take longer than the up-to-date build.

Once we’ve generated both sets of output, we can compare the benchmarked performance. I’ve taken the .csv results from each of those benchmarking executions and combined them into the following table for comparison.

| scenario | default | scenario | default |
| version | Gradle 6.7 | version | Gradle 6.7 |
| tasks | assemble | tasks | clean assemble |
| value | execution | value | execution |
| warm-up build #1 | 14330 | warm-up build #1 | 26492 |
| warm-up build #2 | 1887 | warm-up build #2 | 10345 |
| warm-up build #3 | 1546 | warm-up build #3 | 9853 |
| warm-up build #4 | 1440 | warm-up build #4 | 8757 |
| warm-up build #5 | 1383 | warm-up build #5 | 8705 |
| warm-up build #6 | 1301 | warm-up build #6 | 7377 |
| measured build #1 | 1187 | measured build #1 | 7268 |
| measured build #2 | 1230 | measured build #2 | 7378 |
| measured build #3 | 1118 | measured build #3 | 7750 |
| measured build #4 | 1104 | measured build #4 | 6707 |
| measured build #5 | 1105 | measured build #5 | 6635 |
| measured build #6 | 1082 | measured build #6 | 7542 |
| measured build #7 | 1044 | measured build #7 | 7066 |
| measured build #8 | 990 | measured build #8 | 6398 |
| measured build #9 | 993 | measured build #9 | 6341 |
| measured build #10 | 1037 | measured build #10 | 7416 |

With this, we can see that our up-to-date build takes ~1 second while our clean build is taking ~7 seconds. This seems in line with expectations about the relative performance of these two build types.

Comparing caching impact

We’ve seen the performance of an up-to-date build. We’ve seen the impact of doing a clean build.

Let’s expand on our example by now benchmarking the impact of enabling Gradle’s local build cache.

First, we’ll once again benchmark a clean build without enabling the build cache.

→ gradle-profiler --benchmark --project-dir . clean assemble

Next, we’ll enable Gradle’s local build cache by adding the following to our gradle.properties file:

_org.gradle.caching_=true

Now, we can re-run our clean build benchmark scenario; this time with the cache enabled.

→ gradle-profiler --benchmark --project-dir . clean assemble

Once again, we can compare results to get a sense of how enabling the local Gradle build cache can benefit build performance.

From these results, we see that enabling local caching seems to improve the performance of clean builds from ~7 seconds to ~4.5 seconds.

Now, these build times are artificially fast as it’s a simple project.

However, this approach is applicable to your real-world projects, and these general results are what one might expect from a well-configured project; up-to-date < clean with caching < clean w/out caching.

Configuring Benchmarking Behavior

When running these benchmarking tasks, you may find yourself wanting greater control over how benchmarking is carried out. A few of the common configuration properties you may find yourself needing to change include

• changing the project directory
• modifying the output directory
• updating the number of build iterations

We’re going to quickly examine how you can update these properties when executing your benchmarking task.

Changing Project Directory

Throughout these examples, we’ve been operating as if we are running gradle-profiler from within the project directory.

Here, we see that after the --project-dir flag, we pass . to signify the current directory.

→ gradle-profiler --benchmark --project-dir . assemble

If we want to run gradle-profiler from any other directory, we are free to do that. We just need to update the path to the project directory.

In this updated example, we change directories into our user directory, and pass Projects/GradleProfilerSandbox as the path to our project directory.

→ cd
→ gradle-profiler --benchmark --project-dir Projects/GradleProfilerSandbox/ clean assemble

Changing Output Directory

When gradle-profiler is run, by default, it will store output artifacts in the current directory. If you would prefer to specify a different output directory, you can do so using --output-dir.

gradle-profiler --benchmark --project-dir Projects/GradleProfilerSandbox/ --output-dir Projects/benchmarking/ clean assemble

This may come in handy if you want to automate the running of benchmarking tasks and would like to keep all your outputs collected within a specific working directory.

Controlling Build Warm-Ups and Iterations

Another pair of useful configuration options are --warmups and --iterations. These two flags allow you to control how many warm-up builds and measured builds to run.

This might be useful to you if you want to receive your results more quickly, or if you want more data points, and hopefully greater confidence, if your benchmarking results.

If we wanted to have 10 warm-up builds and 15 measured builds, we can start our benchmarking task like this.

→ gradle-profiler --benchmark --project-dir . --warmups 10 --iterations 15 assemble

Benchmarking Complex Gradle Builds

We’ve really only scratched the surface of what gradle-profiler is capable of. Real world build scenarios are varied, and often quite complex. Ideally, we’d be able to capture these complexities, and benchmark Gradle build performance for these real-world conditions.

Let’s take a look at how we can define benchmark scenarios that give greater control and flexibility into what is benchmarked.

Defining a Benchmark Scenario

As our command line inputs become more complex, it may become difficult to manage.

Look at the following command.

gradle-profiler --benchmark --project-dir Projects/GradleProfilerSandbox/ --output-dir Projects/benchmarking/ --warmups10 --iterations 15 clean assemble

This command is still executing a fairly simple benchmarking task, but has already become quite long and unwieldily to execute from the command line every time.

This is especially true if we want to start benchmarking multiple build types, or want to simulate complex incremental build changes.

To help define complex build scenarios, the gradle-profiler tool provides a mechanism for encapsulating all of the configuration for a build scenario into a .scenarios file. This helps us organize our scenarios in a single place and makes it easier to benchmarking multiple scenarios.

To define a simple .scenarios file, we’ll do the following.

• First, we’ll create a new file named benchmarking.scenarios.
• Next, we’ll define a scenario for our up-to-date assemble task.

assemble_no_op {
  tasks = ["assemble"]
}

In this small configuration block, we’ve defined a scenario named assemble_no_op that will run the assemble task when executed. We’ve used the “no op” suffix on the scenario name because this will test our up-to-date build in which tasks shouldn’t have to be re-run each time.

Benchmarking With a Scenarios File

With our scenario file defined, we can benchmark this scenario by using the --scenario-file flag.

gradle-profiler --benchmark --project-dir . --scenario-file benchmarking.scenarios

From this, we get an HTML output similar to this.

We can see that the assemble_no_op name used to define our scenario is automatically used as the scenario name in the output.

In the next sections, we’ll see how to changes this to something more human-readable and why this can be important.

Configuring Build Scenarios

Within our .scenarios file, there are quite a few configuration options we can use to control our benchmarked scenarios.

We can provide a human-readable title for our scenario:

assemble_no_op {
  title = "Up-To-Date Assemble"
  tasks = ["assemble"]
}

We can provide multiple Gradle tasks to run:

assemble_clean {
  title = "Clean Assemble"
  tasks = ["clean", "assemble"]
}

You could pass in different Gradle build flags such as for parallel execution or for the build cache:

assemble_clean {
  title = "Clean Assemble"
  tasks = ["clean", "assemble"]
  gradle-args = ["--parallel"]
}

We can also explicitly define the number of warm-ups and iterations so they don’t have to be passed from the command line:

assemble_clean {
  title = "Clean Assemble"
  tasks = ["clean", "assemble"]
  gradle-args = ["--parallel"]
  warm-ups = 3
  iterations = 5
}

This is not an exhaustive list of scenario configurations. You can find more examples in the gradle-profiler documentation.

Benchmarking Multiple Build Scenarios

One of the primary benefits of using a .scenarios file is that we can define multiple scenarios within a single file, and benchmark multiple scenarios at the same time.

For example, we could compare our up-to-date, clean, clean w/caching builds from a single benchmarking execution, and receive a single set of output reports comparing all of them.

To do this, we first define each of our unique build scenarios within our benchmarking.scenarios file.

assemble_clean {
  title = "Clean Assemble"
  tasks = ["clean", "assemble"]
}

assemble_clean_caching {
  title = "Clean Assemble w/ Caching"
  tasks = ["clean", "assemble"]
  gradle-args = ["--build-cache"]
}

assemble_no_op {
  title = "Up-To-Date Assemble"
  tasks = ["assemble"]
}

Then we continue to invoke gradle-profiler in the same way; by specifying the single .scenarios file.

gradle-profiler --benchmark --project-dir . --scenario-file benchmarking.scenarios

From this, we will receive a merge report that compares the performance of each scenario.

Scenarios will be run in alphabetical order based on the name used in the scenario definition; not the human-readable title.

When viewing a report with multiple scenarios, you can select a scenario as the baseline. This will update the report to display a +/-% on each build metric where the +/-% is the statistical difference between the current scenario and the baseline scenario.

What does that look like in practice?

In this example, we’ve selected the clean build w/out caching scenario as our baseline.

With the baseline set, we see that the mean build time was reduced ~4% for the clean build with caching scenario and ~80% for the up-to-date build scenario.

By setting a baseline, you let the tool do all the statistical analysis for you leaving you free to interpret and share the results.

Benchmarking Incremental Gradle Builds

The last concept we’re going to touch on is that of benchmarking incremental builds. These are builds in which we need to re-execute a subset of tasks because of file changes. These files could change due to source file changes, resource updates, etc.

Incremental scenarios can be very important for real-world benchmarking of Gradle build performance, because in our day-to-day work, we’re often performing incremental builds as we make a small code change, and redeploy for testing.

When using gradle-profiler we have quite a few options available to us for defining and benchmarking incremental build scenarios.

If building with Kotlin or Java, we might be interested in:

• apply-abi-change-to
• apply-non-abi-change-to

If building an Android application, we might use:

• apply-android-resource-change-to
• apply-android-resource-value-change-to
• apply-android-manifest-change-to
• apply-android-layout-change-to

With these options, and others, we can simulate different types of changes to our projects to benchmark real world scenarios.

Here, we’ve defined a new incremental scenario in our benchmarking.scenarios file.

incremental_build {
  title = "Incremental Assemble w/ Caching"
  tasks = ["assemble"]

  apply-abi-change-to = "app/src/main/java/com/goobar/gradleprofilersandbox/MainActivity.kt"
  apply-android-resource-change-to = "app/src/main/res/values/strings.xml"
  apply-android-resource-value-change-to = "app/src/main/res/values/strings.xml"
}

This scenario will measure the impact of changing one Kotlin file, and of adding and modifying string resource values.

The resulting benchmark results look something like this.

As expected, the incremental build was measured to be faster than a clean build, but slower than an up-to-date build.

This is a very simple example. For your production project, you might define multiple different incremental build scenarios.

If you’re working in a multi-module project, you might want to measure the incremental build impact of changing a single module versus multiple modules. You might want to measure the impact of changing a public api versus changing implementation details of a module. There are many things you may want to measure in order to improve the real world performance of your build.

What’s Next?

At this point, hopefully you have a better understanding of benchmarking Gradle build performance using gradle-profiler, and how to start using gradle-profiler to improve the performance of your Gradle builds.

There’s still more that can be done with gradle-profiler to make it more effective for detecting build regressions and to make it easier to use for everyone on your team.

I’ll be exploring those ideas in upcoming posts.

If you’d like to start learning more today, be sure to check out gradle-profiler on GitHub and Tony Robalik’s post on Benchmarking builds with Gradle-Profiler.

You can find more Gradle-related posts here on my blog.

The post Benchmarking Gradle Builds Using Gradle-Profiler appeared first on goobar.