One Simple Trick For Resolving Java Runtime Dependency Issues

Introduction

If you work with any JVM-based language, such as Java, Kotlin, Scala, Groovy, Clojure etc., you will most likely have come across build and dependency management tools such as Ant / Ivy, Maven, sbt, Leinengen or Gradle.

Fundamentally, the purpose of these tools is to automate your application's build (and sometimes even release) process. They take care of compiling your code, running tests, generating documentation pages (Javadocs) and producing distributable artifacts such as JAR files.

Amongst the main responsibilities and benefits of using a build automation tool is its ability to declaratively manage dependencies for you. For small projects, it is feasible to not use a build system at all, opting instead for a simple shell script that uses javac and jar to compile and package the application.

This becomes tedious and less maintainable once you start adding third-party dependencies to your project. Finding the latest version of the JAR file for the dependency, using that as part of your build process, and including it in the classpath for both compilation and at runtime, whilst also being mindful not to keep unnecessary binaries in your version control system, is a a lot to manage manually.

Although the major build tools within the Java ecosystem provide a declarative way to specify dependencies and handle the process for you, they are not infallible. Occasionally you can end up with problems caused by the inclusion of dependencies that, on the surface, don't make sense, with no obvious cause or solution.

This article aims to explore a problematic dependency management scenario, why it occurs and how to resolve it.

Sample Project

For illustrative purposes, I will be using a Java project with Maven as a minimal working example throughout this article. However, the principles are also applicable to Gradle and other JVM languages. Without further ado, let's dive right in!

Here are the initial contents of the <project> element in pom.xml:

<groupId>org.example</groupId>
<artifactId>minimal</artifactId>
<version>1.0-SNAPSHOT</version>
<name>Dependency Example</name>
<packaging>jar</packaging>

<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>

<dependencies>
<dependency>
    <groupId>com.vonage</groupId>
    <artifactId>client</artifactId>
    <version>7.10.0</version>
</dependency>
</dependencies>

As you can see, we're declaring one dependency - the Vonage Java SDK. Here is our main class, which uses the Account API to obtain our balance, and Messages API to text us this balance.

package com.example.minimal;

import com.vonage.client.VonageClient;
import com.vonage.client.messages.sms.SmsTextRequest;

public class BalancePrinter {
    public static void main(String[] args) throws Throwable {
        VonageClient client = VonageClient.builder()
                .apiKey(System.getenv("VONAGE_API_KEY"))
                .apiSecret(System.getenv("VONAGE_API_SECRET"))
                .build();

        var balance = client.getAccountClient().getBalance();
        var balanceText = SmsTextRequest.builder()
                .from("Vonage").to(System.getenv("TO_NUMBER"))
                .text("Balance: €" + balance.getValue()).build();
        var response = client.getMessagesClient().sendMessage(balanceText);
        System.out.println(response.getMessageUuid());
    }
}

When you run it from your IDE, this works fine. Now let's create a runnable JAR file with all of the dependencies so that we have a standalone executable. We can do this using the Maven Shade plugin:

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.5.0</version>
        <configuration>
          <transformers>
            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
              <mainClass>com.example.minimal.BalancePrinter</mainClass>
            </transformer>
          </transformers>
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

When we run mvn clean install, we get exactly that: an executable standalone JAR (this is located in target/minimal-1.0-SNAPSHOT.jar). And it works! However, did you notice the warnings generated when you ran the mvn command? In the end, you will see something like this:

"[WARNING] maven-shade-plugin has detected that some files are present in two or more JARs. When this happens, only one single version of the file is copied to the uber jar. Usually this is not harmful and you can skip these warnings, otherwise try to manually exclude artifacts based on mvn dependency:tree -Ddetail=true and the above output."

What is this warning trying to tell us? Well, the Java SDK itself has dependencies - you can browse them in the "Runtime Dependencies" section on mvnrepository.com. Those dependencies in turn have their own dependencies and so on.

Since we are creating a single JAR file that "flattens" this dependency hierarchy, we end up with a lot of class files. However, some of these dependencies have dependencies on the same artifact. This process of recursively unpacking each JAR so that there are only classes left means that if there are multiple versions of the same dependency, only one of the classes is retained. After all, it is not possible to have two files with the same name in the same directory.

For example, 'jackson-datatype-jsr310' has a dependency on 'jackson-core', which we also depend on through 'jackson-databind'. Somewhere in the chain are also dependencies on SLF4J logging framework, which is used by many popular libraries. Maven highlights all of the classes in which there are duplicates when flattening the dependency hierarchy.

Where Problems Arise

As the warning states, usually these are not a problem. If it were, then it would be very difficult, if not impossible, to automate building even the simplest projects with dependencies, such as the one in this article.

The strategy for selecting which version of the class to use (i.e. which dependency to pick from) depends on class loading order and details beyond the scope of this article, but the point is that ultimately there can only ever be one instance of a fully qualified class in any given runtime instance of a Java application. When multiple versions are needed, the Shade plugin has a configurable relocation strategy so they can co-exist.

With that in mind, when is this problematic? If by default everything works, when does it fail? To demonstrate, let's return to our example.

Now, let's break it by adding the following dependency to the <dependencies> section of our pom.xml:

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.0-alpha1</version>
</dependency>

If you rebuild with mvn clean install and then try to run the program using the JAR file, you will get the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/conn/HttpClientConnectionManager

This is because you are now overriding one of the core dependencies of the SDK with a much older version which does not contain a class that we depend upon. The same is true if you re-run it from the IDE. You could instead try a different dependency. For example:

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-core</artifactId>
    <version>2.0.0</version>
</dependency>

This will generate the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/util/JacksonFeature

Even though the Vonage Java SDK has no direct dependency on 'jackson-core' it still causes an issue because we are overriding the version of the library (and thus, which class files) are included in our application.

Runtime vs Compile

You may be wondering, how is it that our program managed to build fine despite this erroneous dependency, yet we only came across the issue when running it? This is due to the way dependencies are resolved at compile-time compared to runtime.

At compile-time, we already have all the dependencies we need. After all, the Vonage Java SDK has already been built and proven to work, so we don't need to recompile the classes, we are loading it from an existing JAR. However, at runtime, we substitute the classes used during execution.

The class files have already been generated during the compilation phase, but different versions of those classes may be present at runtime on the classpath.

This is the key to understanding issues with dependencies at runtime: it's almost always due to the classpath being misconfigured. Build systems like Maven and Gradle have sensible defaults and configuration options to help with this, but misusing them as shown above can lead to errors. It's worth understanding the different scopes for dependency declarations.

By default in Maven, a dependency is scoped to be included at both runtime and compile-time - called compile. You can read more about scopes on the official documentation. Similarly in Gradle, the default scope is implementation (as opposed to, e.g. compileOnly).

Although the terminology differs, dependency scoping and propagation are important concepts to understand when relying on any build system to manage your dependencies and ultimately the classpath of your application.

The Simple Solution

The title of this article promised a quick fix for dependency issues. The reality is that, although the problem ultimately has to do with a misconfigured classpath, the fix will depend largely on the complexity of your project and its dependencies.

The mvn dependency:tree command (and similar in other build tools) can help you identify nested and potentially conflicting dependencies, but ultimately it is up to you to ensure that you are not overriding the dependencies used by other libraries you depend upon, as this can lead to conflicts.

Sometimes, the interactions between certain dependencies can cause issues, so the first point of call is to figure out what dependencies they have in common and whether there is a version discrepancy. Is there a version that you can use which is compatible with all of your other dependencies? If so, declare it explicitly.

However, the best way to minimise conflicts in the first place is to only declare dependencies that you use directly. It's usually best practice to minimise the number of declared dependencies in your project and only use what you need.

The more dependencies you add to your project, the more bloated your final artifact becomes and the more potential issues may arise from dependency conflicts. Not to mention, dependencies can reduce your application's security, often through nested dependencies.

For instance, let's say you rely on a library (Library A) which indirectly depends on an older version of another library (Library B) with a security flaw. If Library A's maintainers don't update to B's latest version, that flaw might also impact your application. You can attempt to address this by directly specifying the latest version of Library B in your pom.xml or build.gradle. However, this approach doesn't guarantee compatibility, and it shifts the responsibility from Library A's maintainers to YOU.

Most of the time, as a developer, you neither know nor care about all of the classes included in your application through transitive dependencies. That is, until something breaks.

Then, you have to investigate the classpath. Some tools can help. For example, Maven can provide an "effective pom" using the eponymous plugin. This will give you a clearer picture of what settings are being used in your build (useful for complex projects).

But perhaps the most useful is to understand your dependency graph. Here, the mvn dependency:tree command gives a clear picture of all the libraries and their scope. Look for those with the runtime scope and see if there are any potential conflicts. Other build tools have similar mechanisms. For example, gradle dependencies in Gradle and dependencytree in Ivy. You can then use this information to see where there may be conflicting dependencies.

Signing off

That's all for now! I hope this article has been informative and useful to you. The key takeaway is that dependency issues at runtime are the result of a misconfigured classpath, which means that either something is missing, or the wrong version of it is being used. Understanding the scoping of your dependencies and potential conflicts between transitive dependencies will help you get to the bottom of it.

If you have any comments or suggestions, feel free to reach out to us on X, formerly known as Twitter or drop by our Community Slack. If you enjoyed it, please check out my other Java articles.