Tony Robalik

Posted on Jul 13, 2021 • Edited on Jul 16, 2021

Tools of the build trade: The making of a tiny Kotlin app

#kotlin #gradle #shadow #proguard

Sometimes you need to solve a problem, and you really don't feel like doing it with bash. Recently I took it upon myself to replace an intelligent-but-expensive Gradle task with dumb-but-cheap JVM app.¹ The Gradle task took an average of 2min per build, and my team cumulatively spent 43 hours over the past month waiting for it to complete. The Kotlin app that replaces it takes about 300ms to complete and I project we'll spend the next month cumulatively waiting on it less than 8 minutes. Since time is money, I've estimated this might recoup $100,000 in lost developer productivity over the next 12 months.

This is not that story.²

It is instead the story of how to build such a thing with Gradle. We'll learn how to use the application and distribution plugins to build the app and bundle a distribution; how to use the shadow plugin to turn it into a fat jar; and how to use Proguard to minify the whole thing. The upshot is we turn a 1.5M jar into a 12K "fat" jar, shaving 99.2% off the final binary.

As soon as I saw that number, 99.2, I remembered that Jake Wharton had written about a very similar experience nearly a year ago. His post is a good read. Mine differs in that I will be explaining, step by step, how to achieve similar results with Gradle.

All the code for this is on Github. (Github drop ICE.)

This project is built with Gradle 7.1.1. This is important to keep in mind, as you would need to make some code changes if you were using Gradle 6.x.

The app

This post isn't about the app itself, but we need something to build, so …³

// echo/src/main/kotlin/mutual/aid/App.kt
package mutual.aid

fun main(args: Array<String>) {
  val echo = args.firstOrNull() ?: "Is there an echo in here?"
  println(echo)
}

The real code

One thing I like to stress to people is that build engineering is its own proper domain. I do this because build engineering is how I make money. Let's look at the build code!

Building an app with Gradle

The application plugin makes this very easy.

// echo/build.gradle
plugins {
  id 'org.jetbrains.kotlin.jvm' version '1.5.20'
  id 'application'
}

group = 'mutual.aid'
version = '1.0'

application {
  mainClass = 'mutual.aid.AppKt'
}

We can now build and run this little app with

$ ./gradlew echo:run
Is there an echo in here?

...or if we want to customize our message...

$ ./gradlew echo:run --args="'Nice weather today'"
Nice weather today

We can run our program and provide any argument we want. This is the most fundamental building block for any JVM app.

Turning the app into a distribution

Assuming that you want other people to actually run your app, you should bundle it as a distribution. Let's use the distribution plugin for that.

$ ./gradlew echo:installDist
$ echo/build/install/echo/bin/echo "Unless you're out West, I hear it's pretty hot out there"
Unless you're out West, I hear it's pretty hot out there

(Happily, with this version, we can drop the very un-aesthetic --args=... syntax.)

I'm being a little tricksy. We don't have to apply any new plugins, because the application plugin already takes care of applying the distribution plugin. The latter adds a task, installDist, which installs the distribution into your project's build directory. Here's what the full distribution looks like:

$ tree echo/build/install/echo
echo/build/install/echo
├── bin
│   ├── echo
│   └── echo.bat
└── lib
    ├── annotations-13.0.jar
    ├── echo-1.0.jar
    ├── kotlin-stdlib-1.5.20.jar
    ├── kotlin-stdlib-common-1.5.20.jar
    ├── kotlin-stdlib-jdk7-1.5.20.jar
    └── kotlin-stdlib-jdk8-1.5.20.jar

We can see that it has gathered all of the jars that are on our runtime classpath, including the new jar we've just built, echo-1.0.jar. In addition to these jars, we have two shell scripts, one for *nix and one for Windows. These scripts use the same template that Gradle uses for gradlew[.bat], so they should be pretty robust.

We're already starting to see the problem. We have a "tiny" app, but it still drags along the full Kotlin runtime, despite using very little of it. That lib directory clocks in at 1.7M. All to echo a string! Not only that, but it just seems a little annoying to have all these individual files when all we really want is our program (the jar), and a script to easily invoke it from the command line.

Layering on the Shadow plugin

They say that when you have a problem and you decide to solve it with regex, you now have two problems. The Shadow plugin really ups the ante on that: judging from this article by Alec Strong, when you try to solve a problem by shading, you now have at least five problems.

To be clear, I am joking. John Engelman, the maintainer of the Shadow Gradle Plugin, has done the community a service by making this tool available, free and open source. If the Shadow plugin is hard to use, it's because it's solving a hard problem.

// echo/build.gradle
import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar

plugins {
  id 'org.jetbrains.kotlin.jvm' version '1.5.20'
  id 'application'
  id 'com.github.johnrengelman.shadow' version '7.0.0'
}

def shadowJar = tasks.named('shadowJar', ShadowJar) {
  // the jar remains up to date even when changing excludes
  // https://github.com/johnrengelman/shadow/issues/62
  outputs.upToDateWhen { false }

  group = 'Build'
  description = 'Creates a fat jar'
  archiveFileName = "$archivesBaseName-${version}-all.jar"
  reproducibleFileOrder = true

  from sourceSets.main.output
  from project.configurations.runtimeClasspath

  // Excluding these helps shrink our binary dramatically
  exclude '**/*.kotlin_metadata'
  exclude '**/*.kotlin_module'
  exclude 'META-INF/maven/**'

  // Doesn't work for Kotlin? 
  // https://github.com/johnrengelman/shadow/issues/688
  //minimize()
}

Speaking of hard problems, we can see that I ran into a couple of issues while iterating on this. Feel free to upvote if you want to make my life easier!

I think the most interesting bits are the from and exclude statements. from is telling shadow what to bundle: our actual compilation output, plus the runtime classpath. The exclude statements are important for shrinking our fat jar.

We can already run this fat jar and verify that it still works (the runShadow task is added by the shadow plugin due to its integration with the application plugin):

$ ./gradlew echo:runShadow --args="'I don't mind billionaires going into space, but maybe they could just stay?'"
I don't mind billionaires going into space, but maybe they could just stay?

And finally we can inspect the fat jar itself (this task is also run implicitly when we run the runShadow task):

$ ./gradlew echo:shadowJar
# Produces output at echo/build/libs/echo-1.0-all.jar

If we check its size, we see that it's 1.5M: already down about 12% from the original 1.7M. We can do much better, though.

Layering on Proguard

I know I know. Proguard is so, like, 2019 or whenever it was that R8 came out. But it's free, it's open source, and it has a Gradle task I can configure pretty easily. (And shadow's minify() function didn't work 😭)

The setup:

// echo/build.gradle
import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
import org.gradle.internal.jvm.Jvm
import proguard.gradle.ProGuardTask

buildscript {
  repositories {
    mavenCentral()
  }
  dependencies {
    // There is apparently no plugin
    classpath 'com.guardsquare:proguard-gradle:7.1.0'
  }
}

plugins {
  id 'org.jetbrains.kotlin.jvm' version '1.5.20'
  id 'application'
  id 'com.github.johnrengelman.shadow' version '7.0.0'
}

For reasons that have completely passed me by, Proguard is not bundled as a Gradle plugin, but as a Gradle task, and so to add it to the build script's classpath, I must resort to this ancient black magic.

There is a plugin for Android. And I suspect that someone, somewhere, has created a plugin for JVM projects. But the idea of looking for one that definitely works with Gradle 7 fills me with despair, so I decided to write this blog post instead.

Now we have access to the ProGuardTask task. How do we use it?

// echo/build.gradle
def minify = tasks.register('minify', ProGuardTask) {
  configuration rootProject.file('proguard.pro')

  injars(shadowJar.flatMap { it.archiveFile })
  outjars(layout.buildDirectory.file("libs/${project.name}-${version}-minified.jar"))

  libraryjars(javaRuntime())
  libraryjars(filter: '!**META-INF/versions/**.class', configurations.compileClasspath)
}

/**
 * @return The JDK runtime, for use by Proguard.
 */
List<File> javaRuntime() {
  Jvm jvm = Jvm.current()
  FilenameFilter filter = { _, fileName -> fileName.endsWith(".jar") || fileName.endsWith(".jmod") }

  return ['jmods' /* JDK 9+ */, 'bundle/Classes' /* mac */, 'jre/lib' /* linux */]
    .collect { new File(jvm.javaHome, it) }
    .findAll { it.exists() }
    .collectMany { it.listFiles(filter) as List }
    .toSorted()
    .tap {
      if (isEmpty()) {
        throw new IllegalStateException("Could not find JDK ${jvm.javaVersion.majorVersion} runtime")
      }
    }
}

// proguard.pro
-dontobfuscate

-keep class mutual.aid.AppKt { *; }

That's a mouthful. Since ProGuardTask isn't registered and configured by a plugin, we have to do that ourselves. The first part is telling it our rules, which are very simple; we want to shrink only, and we want to keep our main class entry point. Next, we tell it our injars, which is just the output of the shadowJar task: this is what is getting minified. (Importantly, the syntax I've used means that task dependencies are determined by Gradle without resort to dependsOn.) The outjars function very simply tells the task where to spit out the minified jar. Finally, we have libraryjars, which I think of as the classpath needed to compile my app. These do not get bundled into the output. The most complicated part of this is the javaRuntime() function. The Proguard Github project has a much simpler example, which you can use if you prefer.

Let's run it.

$ ./gradlew echo:minify

If we now inspect echo/build/libs/echo-1.0-minified.jar, we'll see it's only 12K 🎉

Similarly to how we verified that the fat jar was runnable, we can create a JavaExec task and run our minified jar:

tasks.register('runMin', JavaExec) {
  classpath = files(minify)
}

and then run it:

$ ./gradlew echo:runMin --args="'How would space guillotines even work?'"
How would space guillotines even work?

We're not done yet, though. We still need to bundle this minified app as a distribution and publish it.

The making of a skinny-fat distribution

Both the application and shadow plugins register a "zip" task (distZip and shadowDistZip, respectively). Neither are what we want, since they don't package our minified jar. Fortunately, they are both tasks of the core Gradle type Zip, which is easy-ish to configure.

// echo/build.gradle
def startShadowScripts = tasks.named('startShadowScripts', CreateStartScripts) {
  classpath = files(minify)
}

def minifiedDistZip = tasks.register('minifiedDistZip', Zip) { 
  archiveClassifier = 'minified'

  def zipRoot = "/${project.name}-${version}"
  from(minify) {
    into("$zipRoot/lib")
  }
  from(startShadowScripts) {
    into("$zipRoot/bin")
  }
}

The first thing we do is co-opt the startShadowScripts task (what this task does will be explained shortly). Rather than have it use the default classpath (as produced by the shadowJar task), we want it to use our minified jar as the classpath. The syntax classpath = files(minify) is akin to the earlier injars(shadowJar.flatMap { it.archiveFile }) in that it carries task dependency information with it. Since minify is a TaskProvider, files(minify) not only sets the classpath, but also sets the minify task as a dependency of the startShadowScripts task.

Next, we create our own Zip task, minifiedDistZip, and construct it in a way analogous to the base distZip task. Understanding it is easier if we inspect the final product:

$ ./gradlew echo:minifiedDistZip
$ unzip -l echo/build/distributions/echo-1.0-minified.zip 
Archive:  echo/build/distributions/echo-1.0-minified.zip
  Length      Date    Time    Name
--------------  ---------- -----   ----
        0  07-11-2021 17:02   echo-1.0/
        0  07-11-2021 17:02   echo-1.0/lib/
    12513  07-11-2021 16:59   echo-1.0/lib/echo-1.0-minified.jar
        0  07-11-2021 17:02   echo-1.0/bin/
     5640  07-11-2021 17:02   echo-1.0/bin/echo
     2152  07-11-2021 17:02   echo-1.0/bin/echo.bat
--------------                     -------
    20305                     6 files

Our archive contains our fat, minified jar, as well as two scripts, one for *nix and one for Windows. The specific paths are important because the echo and echo.bat scripts that are generated include a CLASSPATH property:

CLASSPATH=$APP_HOME/lib/echo-1.0-minified.jar

There is one little gotcha here that wasted approximately 20 minutes of my life. You may have noticed this

def zipRoot = "/${project.name}-${version}"

That starting / is very important! Without it, the zip contents had very weird paths. Fortunately, I have enough experience with Gradle at this point in my life to know when to take a win and just move on.

Publishing our minified distribution

We now have a zip file with our minified jar, and the zip itself is only 15K, a huge improvement over the original 1.5M zip produced by the distZip task. We have just one more thing to automate, which is publishing this archive. We'll add the maven-publish plugin, and then configure it:

// echo/build.gradle
plugins {
  id 'org.jetbrains.kotlin.jvm' version '1.5.20'
  id 'application'
  id 'com.github.johnrengelman.shadow' version '7.0.0'
  id 'maven-publish'
}

publishing {
  publications {
    minifiedDistribution(MavenPublication) {
      artifact minifiedDistZip
    }
  }
}

This adds a handful of publishing tasks, including publishMinifiedDistributionPublicationToMavenLocal. We can run it and inspect the output:

$ ./gradlew echo:publishMinifiedDistributionPublicationToMavenLocal
$ tree ~/.m2/repository/mutual/aid/echo/
~/.m2/repository/mutual/aid/echo/
├── 1.0
│   ├── echo-1.0-minified.zip
│   └── echo-1.0.pom
└── maven-metadata-local.xml

We even get a pom file, so we could resolve this artifact from a repository by referencing its Maven coordinates mutual.aid:echo:1.0.

Wrapping up

This post is meant as a minimal example of how you might build a small Kotlin app with Gradle, which also had the requirements that it should have zero dependencies, be as small as possible, and be publishable for ease of access by potential users. A lot more is possible if you're willing to read the docs and experiment.

As a reminder, the full source code is available here.

Endnotes

¹ What I mean is that the Kotlin app is very fast because it exploits knowledge of the system, and uses regex to parse build scripts to derive metadata about them. By contrast, the Gradle task is intelligent in that it uses Gradle's own introspection capabilities. Unfortunately, it's 400x slower. Our compromise is to run the task in CI to verify the correctness of the Kotlin app, which is used daily by our developers. up
² I might tell that story in a few months after getting corporate comms approval. Stay tuned! up
³ PRs welcome. up