Tony Robalik

Posted on Aug 21, 2021

Gradle plugins and extensions: A primer for the bemused

#gradle #kotlin

All the code for this project is on Github. (Github drop ICE.)

It's recently occurred to me that not everyone has spent their time as poorly as I have, and therefore may not know how to created custom nested DSLs in Gradle plugins. While surprisingly useful, this is, more importantly, very aesthetic.



// app/build.gradle
theState {
  theDeepState {
    theDeepestState {
      undermine 'the will of the people'
    }
  }
}

Proof that the IDE kinda-almost provides type hints even for good ol' Groovy DSL (more on that in a bit)

Proof that the IDE understands Kotlin DSL better than the Groovy version

Soon after I took those screencaps, I upgraded this project's version of Gradle from 7.1.1 to 7.2, and my IDE (IntelliJ IDEA Ultimate) got confused and no longer gives me DSL hints for Groovy scripts. ¯\_(ツ)_/¯

Leaving aside why we'd want to undermine the will of the people (I mean, isn't it obvious?), how do we do this?

Who this is for

This is for anyone looking for non-trivial examples for one of the fundamental building blocks of Gradle plugin design. I wouldn't go so far as to say they're production-ready (sure as hell I'm not going to be writing any tests!), but I am currently using techniques like these for building a 2+ million LOC application, so…¹

A domain-specific language for the secret government bureaucracy controlling our lives

We'll start by looking at the extension itself, then work backwards to how it is configured and used, and finally how to declare and build it.



// TheStateExtension.kt
package mutual.aid.gradle

import org.gradle.api.Action
import org.gradle.api.Project
import org.gradle.api.model.ObjectFactory
import javax.inject.Inject

open class TheStateExtension @Inject constructor(
  objects: ObjectFactory
) {

  /** Configure the inner DSL object, [TheDeepStateHandler]. */
  val theDeepState: TheDeepStateHandler = objects.newInstance(TheDeepStateHandler::class.java)

  /** Configure the inner DSL object, [TheDeepStateHandler]. */
  fun theDeepState(action: Action<TheDeepStateHandler>) {
    action.execute(theDeepState)
  }

  companion object {
    fun Project.theState(): TheStateExtension {
      return extensions.create("theState", TheStateExtension::class.java)
    }
  }
}

/**
 * An inner DSL object.
 */
open class TheDeepStateHandler @Inject constructor(
  objects: ObjectFactory
) {

  /** Configure the innermost DSL object, [TheDeepestStateHandler]. */
  val theDeepestState: TheDeepestStateHandler = objects.newInstance(TheDeepestStateHandler::class.java)

  /** Configure the innermost DSL object, [TheDeepestStateHandler]. */
  fun theDeepestState(action: Action<TheDeepestStateHandler>) {
    action.execute(theDeepestState)
  }
}

/**
 * An even-more inner-er DSL object.
 */
open class TheDeepestStateHandler {

  private val whoToUndermine = mutableListOf<String>()
  internal val victims: List<String> get() = whoToUndermine.toList()

  /** Tells the app who - or which groups - it should undermine. */
  fun undermine(who: String) {
    whoToUndermine.add(who)
  }
}

Some of the salient points:

I like to name the outermost extension class FooExtension, and the inner DSL objects BarHandler. Having a convention like that makes it easier to navigate in a large code base.
You can inject all of these types with a variety of services (like ObjectFactory), as well as completely arbitrary objects you supply. Just remember to @Inject that constructor!
For something this deep into Gradle territory, let the Gradle APIs do the work for you. Don't try to get creative with Groovy closures or Kotlin lambdas-with-receivers — just use the ObjectFactory and the Action<T> interface. I'll elaborate more on this in a moment.
You can expose the handlers directly (as I have in the example), as well as exposing them via a function, and this lets your users use both dot-notation and DSL-like syntax with curly braces.

Instantiating the extension

Now we know how to create simple inner DSL objects. How do we create and configure the outer-most extension?



// ThePluginOfOppression.kt
package mutual.aid.gradle

import mutual.aid.gradle.TheStateExtension.Companion.theState
import org.gradle.api.Plugin
import org.gradle.api.Project
import org.gradle.api.tasks.JavaExec

class ThePluginOfOppression : Plugin<Project> {

  override fun apply(project: Project): Unit = project.run {
    val theState = theState()
  }
}

Recall that TheStateExtension.theState() is a companion function that is simply project.extensions.create("theState", TheStateExtension::class.java). I like keeping that function with the class itself for encapsulation, as a factory method. It's also important to note that, even though I'm not using the theState instance I create, I still need to create it here so that it can be accessed in a build script when this plugin is applied. Let's go ahead and see how that works, before circling back to actually using the potential user-provided config.

Applying the plugin and configuring the extension in a build script



// app/build.gradle
plugins {
  id 'mutual.aid.oppression-plugin'
}

// 1: DSL-like
theState {
  theDeepState {
    theDeepestState {
      undermine 'the will of the people'
    }
  }
}

// 2: With dot-notation for the laconic
theState
  .theDeepState
  .theDeepestState
  .undermine 'the will of the people'

// 3: Mix and match
theState.theDeepState.theDeepestState {
  undermine 'the will of the people'
}

Easy-peasy. Apply the plugin and configure the extension. Now's a good time to talk about those Action<T> functions that enable the DSL syntax. As a reminder, here's what one looks like:



import org.gradle.api.Action

fun theDeepState(action: Action<TheDeepStateHandler>) {
  action.execute(theDeepState)
}

I keep including the import statements in these code snippets because it's important to note the precise API we're using here — the org.gradle.api API! ² Gradle has special handling for these types. At (build) runtime, Gradle rewrites your build code on the fly (with ASM), such that the method signature theDeepState(action: Action<T>) becomes effectively theDeepState(action: T.() -> Unit). Actually, it is more accurate to say you get both. In my Groovy DSL script, I could also use it. liberally if I preferred (I don't).

Now we know why the IDE struggles with this with its type hints: it sees the source code, which specifies a standard SAM interface; it doesn't see the lambda-with-receiver that is provided on the fly.

It's unclear why it looks better with the Kotlin DSL.³ If you explore the generated type-safe accessors, they also use Action<T>. I guess we'll never know.

Making use of user-provided configuration: who should we oppress today?

Let's go back to our plugin definition, which has now been expanded to use the information provided by our user in our custom DSL.



class ThePluginOfOppression : Plugin<Project> {

  override fun apply(project: Project): Unit = project.run {
    // 1: Apply additional plugins    
    pluginManager.apply("org.jetbrains.kotlin.jvm")
    pluginManager.apply("application")

    // 2: Create our extension
    val theState = theState()

    // 3: Wait for the DSL to be evaluated, and use the information provided
    afterEvaluate {
      tasks.named("run", JavaExec::class.java) {
        it.args = theState.theDeepState.theDeepestState.victims
      }
    }
  }
}

1. Apply additional plugins. It isn't strictly necessary to apply these other plugins by our plugin, but it showcases the versatility of convention plugins, and also helps keep our example more encapsulatd.
2. Create our extension. Same as before.
3. Make use of user-provided data. Sometimes it is not possible to use the Provider API and it's necessary to wait for user data — this is what afterEvaluate was made for. In our case, we're pushing the data — which victims to oppress — into a standard JavaExec task.

Let's run the program and see what happens:



$ ./gradlew -q app:run
Now undermining: the will of the people

Oppression achieved!

Domain object containers

If you're an Android developer, you'll be familiar with this bit of Gradle configuration:



android {
  buildTypes {
    release { ... }
    debug { ... }
    myCustomBuildType { ... }
  }
}

Where do those build types come from? We now know how to generate and use nested DSL objects, but these values are user-provided! The situation becomes slightly more clear when one looks at the Kotlin DSL version of the above:



android {
  buildTypes {
    getByName("release") { ... }
    getByName("debug") { ... }
    create("myCustomBuildType") { ... }
  }
}

buildTypes is a function that provides a NamedDomainObjectContainer<BuildType>. Groovy-flavored Gradle has syntactic sugar that converts debug {} into getByName("debug") {} OR create("debug") {} if that named type has not yet been created. In Kotlin, you have to be explicit. This is also, btw, how I learned that there's no default instance named "release" for signingConfig.

We now know, in rough terms, what a NamedDomainObjectContainer is. How do we create one? How do we get new instances from one? How do we use it? How do our users use it?

Using domain object containers

For this next and final example, let's switch it up. Oppression is boring; how can we help instead?

Let's start with a new extension, ThePeopleExtension:



package mutual.aid.gradle.people

import org.gradle.api.Action
import org.gradle.api.Named
import org.gradle.api.NamedDomainObjectContainer
import org.gradle.api.Project
import org.gradle.api.model.ObjectFactory
import org.gradle.api.provider.Property
import javax.inject.Inject

open class ThePeopleExtension @Inject constructor(objects: ObjectFactory) {

  val problems = objects.domainObjectContainer(ProblemHandler::class.java)

  fun problems(action: Action<NamedDomainObjectContainer<ProblemHandler>>) {
    action.execute(problems)
  }

  companion object {
    internal fun Project.thePeople(): ThePeopleExtension =
      extensions.create("thePeople", ThePeopleExtension::class.java)
  }
}

open class ProblemHandler @Inject constructor(
  private val name: String,
  objects: ObjectFactory
) : Named {

  override fun getName(): String = name

  internal val description: Property<String> = objects.property(String::class.java)
  val solutions = objects.domainObjectContainer(SolutionHandler::class.java)

  fun solutions(action: Action<NamedDomainObjectContainer<SolutionHandler>>) {
    action.execute(solutions)
  }

  fun description(description: String) {
    this.description.set(description)
    this.description.disallowChanges()
  }
}

open class SolutionHandler @Inject constructor(
  private val name: String,
  objects: ObjectFactory
) : Named {

  override fun getName(): String = name

  internal val action: Property<String> = objects.property(String::class.java)
  internal val description: Property<String> = objects.property(String::class.java)
  internal val rank: Property<Int> = objects.property(Int::class.java)

  fun action(action: String) {
    this.action.set(action)
    this.action.disallowChanges()
  }

  fun description(description: String) {
    this.description.set(description)
    this.description.disallowChanges()
  }

  fun rank(rank: Int) {
    this.rank.set(rank)
    this.rank.disallowChanges()
  }
}

Let's talk about a few of the patterns here before we continue.

First, note that the types that are meant to be in a NamedDomainObjectContainer all implement the Named interface. This isn't strictly necessary, but it is necessary that the types have a getName(): String function, or they can't go into a named domain object container.

Second, we create such a container with the method ObjectFactory.domainObjectContainer(Class<T>).

The final interesting pattern in the above is this:



fun description(description: String) {
  this.description.set(action)
  this.description.disallowChanges()
}

As description is a Property<String>, I prefer to keep those values internal and expose them via a function. Users then have a nice DSL like description 'my description (in Groovy), or description("my description") (in Kotlin). Encapsulating those fields also lets me to do extra stuff like call disallowChanges(), which I think is important to prevent violating the principle of least astonishment. Without that, users could call the description() method repeatedly, from multiple locations, and it would be hard to tell where the data were really coming from. When we do this, and someone attempts to call the method more than once, the build will fail.

Let's continue. How does this DSL look "in action"?



// app/build.gradle
thePeople {
  problems {
    climateChange {
      description 'There is no question of cost, because the cost of doing nothing is everything.'
      solutions {
        cleanEnergy {
          description 'We cannot burn any more fossil energy'
          action 'Replace all fossil sources with clean solutions like wind, solar, and geothermal'
          rank 1
        }
        massTransit {
          description 'Single-occupant vehicles are a major source of carbon pollution'
          action 'Increase density in urban environments and build free public transit for all'
          rank 2
        }
        stopEatingAnimals {
          description 'Animal agriculture is one of the top contributors to carbon pollution'
          action 'Most people can thrive on a plant-based diet and do not need animal protein, and could make such a choice with immediate effect'
          rank 3
        }
        antiRacism {
          description 'People of Western European descent (\'white people\') have been the primary beneficiaries of burning fossil carbon'
          action 'White people should should bear the responsibility of paying for climate change mitigation'
          rank 4
        }
        seizeGlobalCapital {
          description 'The costs of climate change are inequitably distributed'
          action 'The costs of climate change mitigation should be born primarily by the wealthiest'
          rank 5
        }
        lastResort {
          description 'If the rich and the powerful refuse to get out of the way of legislative reforms of the system killing us all, there is, unfortunately, always a last resort'
          action 'It starts with \'g\' and rhymes with \'poutine\''
          rank 6
        }
      }
    }
  }
}

I think that's fairly readable, given the complexity of the domain we're trying to model.

But now how do we react to that in our plugin? As always, I think learning is best done by example, so let's tie it all together by looking at how a new plugin, ThePluginOfThePeople, configures tasks based on this user-provided data.



class ThePluginOfThePeople : Plugin<Project> {

  override fun apply(project: Project): Unit = project.run {
    val thePeople = thePeople()

    thePeople.problems.all { problem ->
      tasks.register("listSolutionsFor${problem.name.capitalize()}", ListSolutionsTask::class.java) {
        it.problem.set(problem)
      }
    }
  }
}

abstract class ListSolutionsTask : DefaultTask() {

  init {
    group = "People"
    description = "Prints list of solutions for a given problem"
  }

  @get:Input
  abstract val problem: Property<ProblemsHandler>

  @TaskAction fun action() {
    val problem = problem.get()

    val msg = buildString {
      appendLine(problem.name.capitalize())
      appendLine(problem.description.get())
      appendLine()
      appendLine("Solutions:")
      problem.solutions.sortedBy { it.rank.get() }.forEachIndexed { i, sol ->
        appendLine("${i + 1}. ${sol.name}")
        appendLine("   ${sol.description.get()}")
        appendLine("   ${sol.action.get()}")
      }
    }

    logger.quiet(msg)
  }
}

We can see all the tasks that our new plugin registered quite easily:



$ ./gradlew app:tasks --group people -q

------------------------------------------------------------
Tasks runnable from project ':app'
------------------------------------------------------------

People tasks
------------
listSolutionsForClimateChange - Prints list of solutions for a given problem

In our plugin, we use thePeople.problems.all(Action<T>) to react to user-provided configuration. all(Action<T>) executes the provided action against all elements of the given collection, as well as all future elements that may be added; in this sense, it is lazy. For us, it is useful because the plugin's apply() method runs immediately when the plugin is applied (in the plugins block), which means that the user data is not yet available to react on. all() elegantly solves this problem without recourse to, say, afterEvaluate.

Within our problems.all block, we register a single task — one task per problem — and configure that task by setting its one input as the given ProblemHandler, on a Provider<ProblemHandler>. This is fully serializable, and so is a valid @Input property, as well as being compatible with the experimental configuration cache.

Our task definition is straightforward. It's an abstract class, letting us used managed types (our @Input abstract val problem), and has a simple action. The biggest footgun here is remembering to call get() on the various Provider<String> instances, else we'll get funny output like property 'description$fancy_plugin'.

Finally, let's run one of the generated tasks like so:



$ ./gradlew app:listSolutionsForClimateChange
Configuration cache is an incubating feature.
Calculating task graph as configuration cache cannot be reused because file 'app/build.gradle' has changed.

> Task :app:listSolutionsForclimateChange
ClimateChange
There is no question of cost, because the cost of doing nothing is everything.

Solutions:
1. cleanEnergy
   We cannot burn any more fossil energy
   Replace all fossil sources with clean solutions like wind, solar, and geothermal
2. massTransit
   Single-occupant vehicles are a major source of carbon pollution
   Increase density in urban environments and build free public transit for all
3. stopEatingAnimals
   Animal agriculture is one of the top contributors to carbon pollution
   Most people can thrive on a plant-based diet and do not need animal protein, and could make such a choice with immediate effect
4. antiRacism
   People of Western European descent ('white people') have been the primary beneficiaries of burning fossil carbon
   White people should should bear the responsibility of paying for climate change mitigation
5. seizeGlobalCapital
   The costs of climate change are inequitably distributed
   The costs of climate change mitigation should be born primarily by the wealthiest
6. lastResort
   If the rich and the powerful refuse to get out of the way of legislative reforms of the system killing us all, there is, unfortunately, always a last resort
   It starts with 'g' and rhymes with 'poutine'

Wrapping up

In this post, we learned how to use Gradle to model a complex domain with a nested domain-specific language, or DSL, and how to accommodate custom user data in such a DSL using the NamedDomainObjectContainer. I encourage you to explore the complete sample on Github, which includes build scripts and project layout decisions that were left out of this post, for simplicity.

autonomousapps / gradle-nested-dsl

Endnotes

¹ Watch this space for me lamenting being fired for my terrible technique. up
² I highly recommend exploring these packages in depth. up
³ Jetbrains is the creator of the Kotlin language. up

Top comments (9)

Martin Bonnin • Aug 25 '21

As description is a Property, I prefer to keep those values internal and expose them via a function.

That, + the disallowChanges is very nice 👍.

Do you have any recommendation for properties that can also be outputs of the task and need to be readable too so that they can be wired to other tasks? Would description(): Provider<String> do here? Feels like duplicating a bit the original Property<String> but maybe it's not too bad? Or is there another way to look at this?

Tony Robalik • Aug 25 '21

For sharing information between tasks, I rely entirely on writing output to disk. I don't think any of Gradle's @Output... annotations support non-File types. In your case, I would just write the String to disk in a file that was pointed to by @get:OutputFile abstract val output: RegularFileProperty and then use that output as a file input to the other task. You could write it to disk however you like, but I normally use moshi + json.

Martin Bonnin • Aug 25 '21

Yep, sorry String was a bad example. The question was about exposing internal Properties.

Assuming you have an @get:OutputFile val outputFile: RegularFileProperty, how do you forward that to other parts of the build? I find it somewhat convenient to expose the outputFile itself which then makes it possible to set it without going through the functions and disallowChanges. Maybe not a huge problem but I bumped into this lately and wasn't sure if there was a more idiomatic way.

Tony Robalik • Aug 25 '21

I think that disallowChanges() is more for user-facing API. I do what you're suggesting all the time. E.g., in a task configuration block:

consumerTask.someInputFile.set(producerTask.flatMap { it.someOutputFile })

Those properties aren't meant to be used by users, so if they mess with the properties, that's on them, Undefined Behavior.

Martin Bonnin • Aug 26 '21

I see, thanks!

Grzegorz Antoniak • Jun 16 '22

Really nice post. Thanks for writing it!

El • Aug 29 '21 • Edited

I wondered why the IDE could not support a constantly stable code hints / completion as well on Groovy DSL. that pushed me to change to Kotlin DSL without hesitate though the build time is actually longer. ¯_(ツ)_/¯

Btw, rather than Action<T> blocks, do those methodMissing(...) features cause difficulty on IDE support? (For example like below:

github.com/gradle/gradle/blob/5ec3...
github.com/gradle/gradle/blob/5ec3...

Miguel Aragues • Jan 17 '22

Thank you so much for this post!

The Gradle docs should point to it as introduction. It's so much clearer than the official documentation, with a thousand ways to do the same thing and outdated and useless samples.

wudongsheng • Jan 11 '22

How to achieve like this structure, and theaction is dynamic.
What I mean is How to achieve gradle's dependencies{} strucure

DEV Community