Moving to deps.edn

I've been using Clojure for ... some time now; I think I started experimenting with it in 2009, possibly earlier. At both Aviso and Walmart I have used, and often fought with, Leiningen, the standard build tool.

Finally, I'm changing over to using clj and deps.edn. As is often the case, the inertia of using the tool you know can get in the way of learning a tool that may just fit your needs better.

deps is another case of Clojure's emphasis on data and simplicity, and on explicitness as a better goal than conciseness.

At its core, deps does two things:

• Build the classpath (resolving dependencies and downloading as needed)
• Start a Clojure instance and execute a function using the classpath

There's a few bells and whistles built on top of these two essentials, such as aliases to adjust the classpath and the executed function and function arguments, such as to launch a test runner.

Most importantly, though, deps's more narrow focus makes it easier to understand what's going on as it executes, and to build new and customized functionality that every project seems to need.

For Lacinia (and my other open source projects), I had a goal:

• Store the version number of the library in one place
• Include that version number in generated documentation

In Leiningen, your basic meta-data (defproject) includes the version number right there, accessible to any plugins. With deps, a version number doesn't really make sense ... remember, we're just putting together the classpath and running some Clojure code.

So, if it comes to building and publishing a JAR, you need to implement some project-level build tools.

You do this by defining a :build alias:

  ;; clj -T:build <command>

  :build {:deps {io.github.clojure/tools.build {:git/tag "v0.7.4" :git/sha "ac442da"}
                 slipset/deps-deploy {:mvn/version "0.2.0"}}
          :ns-default build}}

The -T option alias also includes the current directory as a source, so we can create a build.clj file there:

(ns build
  (:require [clojure.tools.build.api :as b]
            [deps-deploy.deps-deploy :as d]))

(def lib 'com.walmartlabs/lacinia)
(def version "1.1-alpha-7")
(def class-dir "target/classes")
(def jar-file (format "target/%s-%s.jar" (name lib) version))
(def copy-srcs ["src" "resources"])

(defn clean
  [_params]
  (b/delete {:path "target"}))

(defn jar
  [_params]
  (let [basis (b/create-basis)]
    (b/write-pom {:class-dir class-dir
                  :lib lib
                  :version version
                  :basis basis
                  :src-dirs ["src"]
                  :resource-dirs ["resources"]})
    (b/copy-dir {:src-dirs copy-srcs
                 :target-dir class-dir})
    (b/jar {:class-dir class-dir
            :jar-file jar-file}))
  (println "Created:" jar-file))

(defn deploy
  [_params]
  (clean nil)
  (jar nil)
  (d/deploy {:installer :remote
             :artifact jar-file
             :pom-file (b/pom-path {:lib lib :class-dir class-dir})
             :sign-releases? true
             :sign-key-id (or (System/getenv "CLOJARS_GPG_ID")
                              (throw (RuntimeException. "CLOJARS_GPG_ID environment variable not set")))}))

Each function here becomes a runnable command, such as clj -T:build clean. Each of these functions accepts a params map and returns it (these are values provided as command arguments, read as Clojure forms, and assembled into a map of keys and values).

Core to all of this is clojure.tools.build.api/create-basis; a basis is a description of the features of the project, including the classpath with source directories and third party dependencies resolved and expanded. Once you have the basis, other build.api commands are used to generate Jar files and deploy to Clojars.

The above build.clj code is good so far ... a bit of boilerplate, but it is reasonably expressive and not too verbose. It's very easy to see how to customize this for different projects.

But back to my challenge - I want to generate documentation with the right version number. Normally, Codox is invoked via an alias defined in deps.edn:

 ;; clj -Xdev:codox

  :codox
  {:extra-deps {codox/codox {:mvn/version "0.10.7"}}
   :exec-fn codox.main/generate-docs
   :exec-args {:metadata {:doc/format :markdown}
               :name "com.walmartlabs/lacinia"
               :version "1.1-alpha-7"
               :description "Clojure-native implementation of GraphQL"}}}

This adds Codox to the classpath along with the project dependencies, and the dependencies of the :dev alias, and invokes codox.main/generate-docs to do the actual work.

Alas, this approach still requires keeping a second copy of the project version synchronized.

Ideally, there should be a way to inject the true version number, from build.clj, into the :exec-args above, but that goes against he grain of deps; the contents of deps.edn are just data; there's no functions to be called to read the version number from somewhere else.

The next best thing is to have a have a codox build command, defined in build.clj, that calls codox.main/generate-docs.

I did a few iterations on that approach but there are problems: the classpath for -T commands does not include the project's classpath, and Codox needs to be able to load namespaces as part of how it organizes and generates the documentation.

Eventually, I came across the following solution:

• Build a classpath that includes the main dependencies, and Codox
• Build and execute a Java command to run Codox

It ends up looking like this:

(defn codox
  [_params]
  (let [basis (b/create-basis {:extra '{:deps {codox/codox {:mvn/version "0.10.8"}}}
                               ;; This is needed because some of the namespaces
                               ;; rely on optional dependencies provided by :dev
                               :aliases [:dev]})
        expression `(do
                      ((requiring-resolve 'codox.main/generate-docs)
                       {:metadata {:doc/format :markdown}
                        :name ~(str lib)
                        :version ~version
                        :description "Clojure-native implementation of GraphQL"})
                      nil)
        process-params (b/java-command
                         {:basis basis
                          :main "clojure.main"
                          :main-args ["--eval" (pr-str expression)]})]
    (b/process process-params)))

At the top, we build a new basis that includes the project's base dependencies, the :dev alias dependencies, and Codox (and its dependencies). We're going to start a process to run a Java command to execute a Clojure expression - this expression ultimately is what invokes generate-docs - and build.api provides all the heavy lifting related to setting up the classpath and starting processes.

Even though the above code isn't a macro definition, we can still use the Clojure syntax quote (the backtick) to create a snippet of Clojure code that gets included on the command line to the Java process.

So, running clj -T:build codox will invoke the codox function, which in-turn starts a second process running Java with an entirely different classpath.

At the end of the day, we have a short command to generate documentation, and our library's version number appears in just one place, build.clj.

I'm just scratching the surface of what can be accomplished using these techniques, but I'm excited at the possibilities of using real Clojure code to tackle some of our interesting build- and deploy-time requirements.