Clojure Bytecode and Namespace Loading

Let’s investigate what kind of bytecode is generated when Clojure compiles *.clj files. First, let’s start with a simple example: (ns example.hello).

Things to Note

Compilation unit is a namespace
A class loader “namespace__init.class” is created for each namespace
Details are available at http://clojure.org/compilation
During compilation, include the (Clojure code to be compiled) source directory in the classpath
At runtime, include clojure.jar in the classpath in addition to the compiled class files

Compilation

Without using leiningen, let’s compile with raw Clojure. Three class files are created as follows:

hello__init.class
hello$fn__4.class
hello$loading__4958__auto__.class

$ mkdir clojure-work ; cd clojure-work
$ wget http://central.maven.org/maven2/org/clojure/clojure/1.6.0/clojure-1.6.0.jar
$ mkdir -p src/example
$ vim src/example/hello.clj
  (ns example.hello)
$ mkdir classes
$ tree
    .
    ├── classes
    ├── clojure-1.6.0.jar
    └── src
        └── example
            └── hello.clj
$ java -cp ./src:./classes:clojure-1.6.0.jar clojure.main
user=> (compile 'example.hello)
$ tree classes
    classes
    └── example
        ├── hello$fn__4.class
        ├── hello__init.class
        └── hello$loading__4958__auto__.class
## If you define a main function in hello.clj
## To generate hello.class, compile with :gen-class.
# $ java -cp clojure-1.6.0.jar:./classes example.hello

Macro Expansion

Since creating a namespace (ns ’example.hello) is a macro, let’s expand it. First, the result of macroexpand.

(do (clojure.core/in-ns (quote example.hello))
    (clojure.core/with-loading-context (clojure.core/refer (quote clojure.core)))
    (if (.equals (quote example.hello) (quote clojure.core))
      nil
      (do (clojure.core/dosync
            (clojure.core/commute
              (clojure.core/deref (var clojure.core/*loaded-libs*))
              clojure.core/conj
              (quote example.hello)))
          nil)))

Further expansion results in the following. However, macroexpand can’t fully expand it, so I predicted this from reference materials. I’ll write details later, but the whole thing is written in hello__init.class.

(do
  (clojure.core/in-ns (quote example.hello))
  ((fn* loading__4958__auto__
        ([]
         (. clojure.lang.Var (clojure.core/pushThreadBindings {clojure.lang.Compiler/LOADER (. (. loading__4958__auto__ getClass) getClassLoader)}))
         (try
           (clojure.core/refer (quote clojure.core))
           (finally
             (. clojure.lang.Var (clojure.core/popThreadBindings)))))))
  (if (. (quote example.hello) equals (quote clojure.core))
    nil
    (do
      (. clojure.lang.LockingTransaction (clojure.core/runInTransaction (fn*
                                                                          ([]
                                                                           (clojure.core/commute (clojure.core/deref (var clojure.core/*loaded-libs*))
                                                                                                 clojure.core/conj
                                                                                                 (quote example.hello))))))
      nil)))

From here on, let’s look at the generated class files.

hello__init.class

When loading a namespace, it searches for and loads the AOT __init.class file. The static initializer runs when the class is loaded, and loading of functions under the namespace happens at this timing. The following is a Java pseudo-code representation of the decompiled output using javap. pushNSandLoader and popthreadbindings modify Clojure’s top-level bindings like *ns*, *fn-loader*, *read-eval*. (in-ns … ) and (with-loading-context … ) are called using IFn. In particular, (with-loading-context … ) has its implementation in hello$loading__4958__auto__.class.

package example;
import clojure.lang.*;

public class hello__init {
  public static {};                 // static initializer

  public static final Var const__0;
  public static final AFn const__1;
  public static final AFn const__2;

  public static void load();
  public static void __init0();
}

static {
  __init0();
  ClassLoader loader = Class.forName("example.hello__init").getClassLoader();
  clojure.lang.Compiler.pushNSandLoader(loader);
  try {
    load();
  } finally {
    clojure.lang.Var.popThreadBindings();
  }
}

static void __init0() {
  const__0 = (Var)RT.var("clojure.core", "in-ns");
  const__1 = (AFn)Symbol.intern(null, "example.hello");
  const__2 = (AFn)Symbol.intern(null, "clojure.core");
}

public static void load() {
  // (in-ns 'example.hello)
  IFn inNs = (IFn)const__0.getRawRoot();
  inNs.invoke(const__1);

  // (with-loading-context (refer 'clojure.core))
  IFn loading4958auto = (IFn)new example.hello$loading__4958__auto();
  loading4958auto.invoke();

  // (if (.equals 'example.hello 'clojure.core)
  //   nil
  //   (do
  //     (LockingTransaction/runIntransaction (fn* …))
  //     nil))
  Symbol exampleHello = (Symbol)const__1;
  if (exampleHello.equals(const__2)) {
    return null;
  } else {
    Callable callable = (Callable)new example.hello$fn__4();
    LockingTransaction.runInTransaction(callable);
    return null;
  }
}

hello$loading4958auto__.class

Corresponds to (with-loading-context (refer 'clojure-core)). The AFunction class seems to correspond to one Clojure function,

At load time, use static initializer
When using the function, use constructor

mapUniqueKeys is fetching arguments, but I haven’t investigated the details. I kind of get the general idea though.

package example;
import clojure.lang.*;

public final class hello$loading__4958__auto__ extends AFunction {
  public static final Var const__0;
  public static final AFn const__1;

  public static {};
  public hello$loading__4958__auto__();

  public Object invoke();
}

public static {
  const__0 = (Var)Rt.var("clojure.core", "refer");
  const__1 = (AFn)Symbol.intern(null, "clojure.core");
}

public hello$loading__4958__auto__() {
    super();
}

public Object invoke() {
  // (Var/pushThreadBindings {Compiler/LOADER (.getClassLoader (.getClass loading__4958__auto__))})
  Object[] bindings = new Object[2];
  bindings[0] = Compiler.LOADER;
  bindings[1] = this.getClass().getClassLoader();
  Var.pushThreadBindings((Associative)RT.mapUniqueKeys(bindings));

  try {
    // (refer 'clojure.core)
    IFn refer = (IFn)const__0.getRawRoot();
    return refer.invoke(const__1);
  } finally {
    Var.popThreadBindings();
  }
}

hello$fn__4.class

First, let me write the expanded version. It corresponds to (fn* ([] …)). I understand that it adds example.hello to *loaded-libs*.

(dosync (commute (deref #'clojure.core/*loaded-libs*)
                 conj
                 'example.hello))

=>

(LockingTransaction/runInTransaction
  (fn* ([] (commute (deref #'clojure.core/*loaded-libs*)
                    conj
                    'example.hello))))

package example;
import clojure.lang.*;

public final class hello$fn__4 extends AFunction {
  public static final Var const__0;
  public static final Var const__1;
  public static final Var const__2;
  public static final Var const__3;
  public static final AFn const__4;

  public static {};

  public hello$fn__4();

  public Object invoke();
}

public static {
  const__0 = (Var)RT.var("clojure.core", "commute");
  const__1 = (Var)RT.var("clojure.core", "deref");
  const__2 = (Var)RT.var("clojure.core", "*loaded-libs*");
  const__3 = (Var)RT.var("clojure.core", "conj");
  const__4 = (AFn)Symbol.intern(null, "example.hello");
}

public hello$fn__4() {
    super();
}

public Object invoke() {
  IFn commute = (IFn)const__0.getRawRoot();
  IFn deref = (IFn)const__1.getRawRoot();
  // (deref #'clojure.core/*loaded-libs*)
  Object loadedLibs = deref.invoke(const__2);
  Object conj = const__3.getRawRoot();
  // (commute loadedLibs conj 'example.hello)
  return commute.invoke(loadedLibs, conj, const__4);
}

References

How Clojure works: a simple namespace

Things to Note#

Compilation#

Macro Expansion#

hello__init.class#

hello$loading__4958__auto__.class#

hello$fn__4.class#

References#