A crash course in classpaths: Run!

Over the past three posts, we have explored classpaths and class loading on the JVM and Android. Till now, we have focused on the build: we've learned how to inspect, influence, and even manipulate build classpaths to achieve our software goals.

In this post, we're going to take this a step further and look at the classpaths involved in compiling and running our applications themselves. These are distinct from the classpaths that influence our build environment, and are critical for understanding our apps. Knowing how they work will make you a more effective JVM/ART (Android Runtime) programmer; it will help you solve problems and avoid the hyperbolic curve of desperation that is reading ancient answers on Stack Overflow.

By the end of this post, the next time you see a NoClassDefFoundError, you will feel like a superhero rather than an innocent bystander.

Foundations

Let's start with a simple Kotlin program, which we'll use to probe a bit what we mean when we talk about compile-time vs runtime classpaths.

In these examples, we'll be using kotlinc (for compilation of Kotlin source), and kotlin (for running Kotlin programs). (official docs)

// Main.kt
import lib.Library

fun main() {
  Library().truth()
}

// lib/Library.kt
package lib

class Library {
  fun truth() = println("All billionaires are bad.")
}

In order to compile this program, we'll need to ensure all the classes involved are on the compile classpath: both Main and our library, Library. That looks like the following:

$ cd lib && kotlinc Library.kt && cd ..

This produces a class file, lib/Library.class (the path is important). We need this for the next step:

$ kotlinc -classpath lib Main.kt -d main.jar

This compiles Main.kt. The program kotlinc will look for a class file at lib/Library.class (based on the import statement, import lib.Library in Main.kt!). Including -classpath lib ensures that the library class file is available for compiling Main.kt.

Finally, we have -d main.jar, which tells kotlinc to bundle its output into a jar with that name. It is illuminating to inspect the jar that results from this command:

$ jar tf main.jar
META-INF/MANIFEST.MF
META-INF/main.kotlin_module
MainKt.class

We see the usual jar cruft (manifest, kotlin_module); more importantly, we see our compiled class, MainKt.class. Note that one thing we don't see is Library.class. We compiled against it, but it does not get bundled into our jar file. When we run our program, we'll have to provide that class file another way, on the runtime classpath. Let's try to figure out how to do that. We'll start naively:

$ kotlin MainKt
error: could not find or load main class MainKt

Whoops, that didn't work. Let's try again:

$ kotlin -classpath main.jar MainKt
Exception in thread "main" java.lang.NoClassDefFoundError: lib/Library
...

Arrgh. One last time:

$ kotlin -classpath lib:main.jar MainKt
All billionaires are bad.

What's going on here? Well, the first command didn't work because MainKt was not on the classpath for our kotlin invocation (kotlin could not find the class MainKt, which we told it to run). We fixed that by telling kotlin about main.jar, but the command still failed, this time with a NoClassDefFoundError for lib/Library. We then fixed that by updating our classpath to include the directory that contains the library class file (class files and jars are specified on a :-delimited list). This last command finally succeeded, and in an even more positive turn of events, taught us an important truth!

We now have a better grasp of the difference between the compile-time and runtime classpaths. In particular, we know that specifying the compile-time classpath (as we did with kotlinc) is not sufficient to also run our program (with kotlin). For that, we need to provide a separate (but possibly identical) runtime classpath.

The runtime would like a word

What about when the runtime differs from the compile-time? Let's explore.

// Main.kt same as before
// lib/Library.kt same as before

// lib-alt/Library.kt
package lib

class Library {
  fun truth() = println("Some billionaires are very fine people.")
}

We've already compiled Main.kt and lib/Library.kt. Let's compile lib-alt/Library.kt.

$ cd lib-alt && kotlinc Library.kt && cd ..

And now run our program again, but with a different runtime classpath!

$ kotlin -classpath lib-alt:main.jar MainKt
Some billionaires are very fine people.

We did not recompile Main.kt. We simply ran it in another context — a different classpath that provided a different implementation of the truth() function (lib-alt/Library and lib/Library are binary-compatible). We also learned another important truth, albeit one in a meta-ironic mode.

On the perils of being too clever

I want to really hammer home the power of the runtime, so let's work through one more example. We'll start by updating our alternative library:

// lib-alt/Library.kt
package lib

class Library {
  fun truth() = println("Some billionaires are very fine people.")
  fun lie() = println("Actually, billionaires earned their wealth.")
}

Sneaky! I wonder what that's for.

// Main.kt
import lib.Library

fun main(args: Array<String>) {
  val lib = Library()

  val arg = args.firstOrNull() ?: "truth"
  lib.javaClass
    .getDeclaredMethod(arg)
    .invoke(lib)
}

Huh... where's this going? Note that we must recompile Main.kt because we have changed it this time, unlike in our past examples. We also use a new flag, -include-runtime, which bundles the Kotlin stdlib (the "runtime") into our jar, so we can use the stdlib function firstOrNull() without having to manually put it on the classpath later.

$ kotlinc -classpath lib Main.kt -include-runtime -d main.jar
$ cd lib-alt && kotlinc Library.kt && cd ..

Note that the order of compilations above does not matter, nor the fact that we first compile Main.kt against lib/Library, and later run it against lib-alt/Library. The Kotlin compiler only needs some way to resolve all the symbols, and in this case, that is only import lib.Library.

Ok! Time to run our new program.

$ kotlin -classpath lib-alt:main.jar MainKt lie
Actually, billionaires earned their wealth.

See, this is why reflection is bad.

Connecting this to Gradle

Since we don't generally use the SDK tools directly, but rather via a build tool such as Gradle, let's take a moment to translate the above into Gradle terms. Imagine a project with a structure like the following:

.
├── app
│   ├── build.gradle
│   └── src
│       └── main
│           └── Main.kt
├── lib
│   ├── build.gradle
│   └── src
│       └── main
│           └── lib
│               └── Library.kt
├── lib-alt
│   ├── build.gradle
│   └── src
│       └── main
│           └── lib
│               └── Library.kt
└── settings.gradle

You already know what the source files look like. settings.gradle is simple:

include ':app', ':lib', ':lib-alt'

The only other interesting file is app/build.gradle:

plugins {
  id 'org.jetbrains.kotlin.jvm' version '1.5.10'
}
dependencies {
  // Two choices:
  // 1. :lib is on the compile and runtime classpaths of :app
  implementation project(':lib')

  // 2. :lib is on the compile classpath, while :lib-alt is
  //    on the runtime classpath.
  compileOnly project(':lib')
  runtimeOnly project(':lib-alt')
}

Gradle's docs have a nice graphic describing the relationship between the various "configurations" here.

Use-cases

Writing maximally-compatible Gradle plugins

Let's say you want to write a Gradle plugin that is compatible across a range of Android Gradle Plugin (AGP) versions, say 4.2 through 7.0. There are at least two things you should do:

1. Compile your plugin against the minimum version you intend to support. In this case, 4.2.0.
2. Declare this dependency as a compileOnly dependency, not as implementation or api.

Here's what that looks like:

dependencies {
  // 4.2.0 is the minimum version we support
  compileOnly 'com.android.tools.build:gradle-api:4.2.0'
}

Doing this ensures that the dependency is available at compile-time, and also that it is not exposed to users of your plugin at runtime. (For an excellent discussion of Gradle "configurations" like compileOnly, etc., see this post by Martin Bonnin.) Because the dependency is available at compile-time, you can see all its types in your IDE, enabling a pleasant development experience. And because the dependency is not automatically supplied at runtime, you're requiring your users to ensure it's on their build-runtime classpath — which they normally will have anyway, if they're using a plugin (yours) that is meant to enhance AGP in some way.

Mysterious RuntimeExceptions: Stub!

At work, we were seeing unit test failures, but only in the IDE. The command line, and CI, worked fine. (In fact, this is how the IDE error got distributed to the whole company: green build on CI.)

Caused by: java.lang.RuntimeException: Stub!
    at android.os.Looper.getMainLooper(Looper.java:7)
    at flow.Preconditions.getMainThread(Preconditions.java:55)
    at flow.Preconditions.<clinit>(Preconditions.java:22)

Naturally, we were suspicious that this was an IDE bug (spoiler alert: it was), but since app devs generally prefer to run tests from their IDEs, we couldn't just ignore it. After all, my team is named "Mobile Developer Experience," not "It's Green on CI."

If you have a bit of experience with Android, you may already be thinking that we've somehow messed up and are trying to unit test something that ought to be instrument-tested instead. If so, you'd be kind of on to something. We had recently added a compileOnly (!) dependency on 'com.google.android:android:4.1.1.4' — these are the maven coordinates for "android.jar," that is, the Android runtime. We wanted to compile against them because some code touched the Parcelable class, but didn't actually use it. Therefore, we knew we'd be safe at runtime because we'd never try to invoke anything relating to a Parcelable — it just had to be available at compile-time.

To reiterate, this worked as expected from the command line, but when invoking tests from Android Studio, they'd always fail (at least it was deterministic). Turns out the reason was, and I quote (emphasis mine) "there are classpath issues running the tests inside the IDE because the whole setup is a mess," and "unit tests have some many issues before [Arctic Fox], it is surprising they worked at all." Basically, Android Studio before Arctic Fox is using the wrong runtime classpath, leading to runtime failures.

The fix for this turned out to be very simple, once the problem was understood. First, we removed compileOnly 'com.google.android:android:4.1.1.4'. We added a new module which had hand-written stubs for Parcelable and Parcel, and we put that module on the compile classpath instead. Now all our tests pass, from both CLI and (old, broken) Android Studio!

Understanding NoClassDefFoundErrors

Maybe you've seen something like the following in your career as an Android or JVM dev:

Instrumentation test failed due to uncaught exception in thread [main]:
java.lang.NoClassDefFoundError: Failed resolution of: Lcom/example/ThisClassShouldDefinitelyBeHere;

"Huh," you think. "ThisClassShouldDefinitelyBeHere should definitely be there. What's going on?"

What's going on is a runtime failure because the ART/JVM tried to resolve a class and couldn't find it anywhere: it was not available on the runtime classpath.

"But my code compiled fine!" you say. "That class is in dependency 'foo', which is declared on the 'implementation' configuration. Why isn't it available at runtime?"

These are excellent questions. You've already identified that the class should be there: you declared it on the right configuration, implementation, and you know that such dependencies are — or ought to be — available at both compile-time and runtime. So what you've found, actually, is a bug in one of your build tools. I recently encountered just such an issue with AGP 4.2.0, which occurs with project dependencies declared on androidTestImplementation and which contain mixed Java/Kotlin source. The bug is already fixed and will be released as 4.2.2, but we've monkey patched it locally while we wait for the official release.

The takeaway here is that, once you have a sufficient understanding of what can cause problems like NoClassDefFoundError, you can already eliminate a whole class of possible causes and narrow your focus considerably.

Well, that's it for the crash course in classpaths. I hope you've found it interesting and useful. Happy coding!

Thanks once again to César Puerta for his typically thorough reviews. If at any point you find this post confusing, know that it means I ignored César's advice. Thanks as well to Nate Ebel for his early encouragement.