I was recently talking with someone who is learning Clojure, and while learning destructuring he showed me an example function:

(defn my-first [[s]] s)

This function destructures the argument into a sequence, with the first element bound to s. The function then returns s.

Having seen how this shows up in bytecode, I commented that it basically does first internally anyway. However, I haven't looked at the bytecode generated by Clojure for a while, and this was a good opportunity to check my assumptions. It turned out that my claim was wrong, as it actually uses the nth function. This makes sense, as it allows sequence destructuring into multiple values to be consistent for any number of elements, and not just the first one. But the idea that it just gets translated into code you could write yourself is the same.

Getting the Bytecode

My skill with development tools is reasonably poor, as I have a bad habit of finding something that works and sticking with it. There are undoubtedly better techniques than the ones I use. (I'm open to suggestions!)

no.disassemble

My first thought was to use my friend Gary's no.disassemble library. While he was the first to point out its shortcomings, I recall it being quite useful.

The README for this project recommends using lein-nodisassemble plugin for Leiningen to load the library as a plugin. Unfortunately, this does not have a no.disassemble dependency, so I had to make sure that the library was in the dependencies, and the plugin was explicitly listed. Once these were in, I tried disassembling a simple function and immediately got a NullPointerException.

This library was first released in 2013 and hasn't been updated since 2015. We've had several releases of Java and Clojure since then, so it looks like it just went stale.

AOT Compilation

Without a ready-made tool to pull apart the bytecode in a JVM session, I decided to drop back to the most standard method, which is the javap tool. It's possible to use an API based system like BCEL or ASM but since I just want to inspect the data then javap makes sense, as it prints everything out in a readable format.

One limitation to using javap is that it runs against saved class files, which means that it can't pick up dynamically eval'ed Clojure code. That requires that to analyze a function, I need to fully compile it. If a project is already setup, then Leiningen can easily compile a namespace to classes by adding an :aot, or "Ahead Of Time" compiler directive:

:aot [the.namespace]

However, for a simple bit of code analysis, I can use the much faster command line clj to get a Clojure REPL, and run the compiler from there:

=> (compile 'the.namespace)

Example Code

Let's set up a simple namespace:

(ns disassemble.core)

(defn my-first [[f]] f)

(defn my-second [[_ s]] s)

The my-first function destructures a seq into a single element while ignoring the rest of the sequence, and then returns this element. The my-second function destructures a seq into the first and second elements and returns the second. The first element is destructured into the var named _. A common Clojure convention is to use this var name to indicate an unused value, though it is actually a legitimate name for a var and can be referenced.

This namespace should go under the working directory in the file: src/disassemble/core.clj

The Clojure compiler also likes to output to a directory called classes, meaning that this must be made as well. Finally, a deps.edn file is needed to tell the Clojure runtime where to find files in the classpath. It's a good idea to include the classes being generated along with the src directory:

$ mkdir classes
$ echo '{:paths ["src" "classes"]}' > deps.edn

With all of that set up, we can compile from a repl, or directly from the command line:

$ clj -e "(compile 'disassemble.core)"

Now if we look in the classes/dissassemble directory we can see the generated class files:

core$fn__128.class                      core$my_second.class
core$loading__6721__auto____126.class   core__init.class
core$my_first.class

This includes the namespace loader core__init.class and some autogenerated code, but what we are really looking for here are the classes that contain the function implementations: core$my_first.class, core$my_second.class

my-first

We can look at the signature of the class with a simple invocation of javap:

$ javap -cp classes 'disassemble.core$my_first'
Compiled from "core.clj"
public final class disassemble.core$my_first extends clojure.lang.AFunction {
  public disassemble.core$my_first();
  public static java.lang.Object invokeStatic(java.lang.Object);
  public java.lang.Object invoke(java.lang.Object);
  public static {};
}

This contains the constructor, and invokeStatic static method, and an invoke member method. It's a fair guess to think that the invoke method will call invokeStatic, and this is indeed what it does.

Let's see how the function is implemented by using the "disassemble" command line option of -c:

$ javap -cp classes -c 'disassemble.core$my_first'
Compiled from "core.clj"
public final class disassemble.core$my_first extends clojure.lang.AFunction {
  public disassemble.core$my_first();
    Code:
       0: aload_0
       1: invokespecial #9                  // Method clojure/lang/AFunction."<init>":()V
       4: return

  public static java.lang.Object invokeStatic(java.lang.Object);
    Code:
       0: aload_0
       1: aconst_null
       2: astore_0
       3: astore_1
       4: aload_1
       5: aconst_null
       6: astore_1
       7: lconst_0
       8: invokestatic  #17                 // Method clojure/lang/RT.intCast:(J)I
      11: aconst_null
      12: invokestatic  #21                 // Method clojure/lang/RT.nth:(Ljava/lang/Object;ILjava/lang/Object;)Ljava/lang/Object;
      15: astore_2
      16: aload_2
      17: aconst_null
      18: astore_2
      19: areturn

  public java.lang.Object invoke(java.lang.Object);
    Code:
       0: aload_1
       1: aconst_null
       2: astore_1
       3: invokestatic  #28                 // Method invokeStatic:(Ljava/lang/Object;)Ljava/lang/Object;
       6: areturn

  public static {};
    Code:
       0: return
}

This describes 4 sections:

• The constructor disassemble.core$my_first(). This calls the super constructor in the AFunction class which this function class extends.
• The invokeStatic static method, described below.
• The invoke method. This calls invokeStatic and returns whatever that static method returned.
• The static code block, which is empty for this class.

Breaking it Down

Java methods mostly operate on 2 data structures: the Local Variables array and the Operand Stack. Each method sees only its own stack and local variables.

The Local Variables array starts with the arguments that were passed to the method, in the order they were provided. These can be overwritten and reused. The array may also be large enough to hold other data.

The Operand Stack is a last-in-first-out (LIFO) stack. When arguments are to be passed to a method, they are placed here. Once the method returns, those arguments will have been removed, and the result of the method will be sitting at the top of the stack.

Say a method is to be called with 2 arguments, arg1 and arg2, and will return the value ret. The order of operations is:

• The stack may have some data on it, represented by .... This may be anything or may be empty. Stack (...)
• Add arg1 to the top of the stack. Stack: (..., arg1)
• Add arg2 to the top of the stack. Stack: (..., arg1, arg2)
• Call the method.
• At the start of the method, the stack is empty. The local variables array as the arguments. Stack: (); Variables [arg1, arg2]
• The method uses the arguments to determine a result. This is then pushed to the stack. Stack: (res)
• The return operation is executed.
• Control returns to the original function. The stack is now: (..., res)

With this in mind, let's look at the code for invokeStatic. First of all, recall the stack is empty, and the local variables array contains the argument (a sequence) in the first element. I'll refer to local variables with array notation, so the first element is local[0]. This array is 3 elements long, as the compiler figures out the required length ahead of time. If the argument given to this method is x, then the array will look like:
[x, null, null]

0: aload_0
This puts the 0 element of the local variable array (local[0], or the first argument) on the stack.
1: aconst_null
This puts a null onto the stack
2: astore_0
This removes the value at the top of the stack (null) and put it into local[0]
3: astore_1
Put the value on the top of the stack (the first argument) into local[1]

So far, this moved the value of local[0] into local[1] and cleared local[0]. The Local Variables array will now look like: [null, x, null]

4: aload_1
Take the x in local[1] and put it on the stack.
5: aconst_null
6: astore_1
Puts null on the stack and then moves it into local[1]. Now the local variables are all null. The top of the stack is x.
7: lconst_0
The stack is now: (x, 0L)
8: invokestatic #17
This calls the clojure.lang.RT.intCast(long) method. This takes the long 0 value from the top of the stack, and replaces it with an int value instead. The stack is now: (x, 0)

Why do this? The Clojure language defaults to using the long type for integral numbers, so the compiling code would be expecting to convert these values from long whenever an int is needed.

11: aconst_null
Add a null to the top of the stack. The stack is now: (x, 0, null)
12: invokestatic #21
This calls the clojure.lang.RT.nth(Object,int,Object) method, passing the arguments: x, 0, null.

This function treats x as a sequence, and figures out the best way to get the 0 element from it. If you follow the link, you'll see that it uses and index when available, or falls back to several other methods, depending on which is best.

Let's call the thing that it finds f. Upon return, the stack has had the 3 values removed, and now looks like: (f)

At this point, the method could return, as the required result is at the top of the stack. However, the compiler is more general-purpose than this. Consider the Clojure code again:

(defn my-first [[f]] f)

So far, the compiled code has extracted the required value, which is now on the stack. It wants to store this to be accessible as a local value, which is what is referred to in the source code as f. This is done by storing the value into the local variables. The final line will then get that value out and return it. We see that in the next few lines.

15: astore_2
Stores the value into local[2]. The stack is now empty, and the local vars are: [null null f]
16: aload_2
Gets local[2] and puts it back on the stack.
17: aconst_null
18: astore_2
Puts a null onto the stack, and then move it into local[2]. So the local variables are back to all null again, and the stack just has f.
19: areturn
This returns the f value.

Overview

The first 6 instructions shuffle the x argument around, ending up with a stack holding just x. The next few instruction get a int 0 and a null onto the stack, so finally the nth method can be called on x. The result gets temporarily stored, before being returned.

A few things to note...

A lot of work goes into ensuring that the local variables are kept null when they are not in use. This reminds me of secure coding, where data is not kept in memory for any longer than it needs to be. However, for the general case, I'm wondering how essential that is, and if the JIT (Just In Time compiler) can see when this is unnecessary and avoid it. If the JIT doesn't see it, then is there a performance impact if Clojure is doing it everywhere?

Clojure is using a standard pattern to restructure the seq argument that I referred to as x. The expectation would be to use this in the body of a method, so it makes sense to save it into local[2]. However, the unusual nature of this function that simply returns its destructured value isn't something that was really considered (nor should it be) by the compiler.

Also interesting is how Clojure uses long values, but when it needs an int it will start with a long and will convert it by calling the RT.intCast method. That's a bit roundabout. I'm left wondering why RT.nth(Object,int,Object) doesn't just take a long and make the cast itself. (It gets murkier with looking at RT.nth(Object,int) since that's called both inside the Clojure runtime, and by users calling nth).

References

I can never remember most of these operations, so I just look them up. The official document for this is the Java Virtual Machine Specification, the latest version of which is for Java SE 14.

The description of the local variables and stack is provided in the Frames section. The individual opcodes are all in the Instructions section. However, I find this format a little annoying to read. I much prefer the terse table format on Wikipedia.