Predictable Code in Elixir: Expressions as Reducers and Macros

In the first part of this series on maintainable Elixir code, we started by applying rules for code predictability in our code design. We immediately saw an emerging pattern: a series of transformations on state. In this part, we'll explore this pattern further.

We'll first learn how to write expressions as reducers. Then we'll use metaprogramming to make use of reducers and enforce code style seamlessly.

Finishing up, we'll see an example where all the pieces fit together.

Let's get going!

Quick Recap of Part One and What We'll Do

In the first part, we experimented with applying code styles to this snippet:

with %Payment{} = payment <- fetch_payment_information(params),
     {:ok, user} <- Session.get(conn, :user),
     address when !is_nil(address) <- fetch_address(user, params),
     {:ok, order} <- create_order(user, payment, address) do
  conn
  |> put_flash(:info, "Order completed!")
  |> render("checkout.html")
else
  {:error, :payment_failed} ->
    handle_error(conn, "Payment Error")

  %Store.OrderError{message: message} ->
    handle_error(conn, "Order Error")

  error ->
    handle_error(conn, "Unprocessable order")
end

By the end of this post, we'll have all the tools to rewrite it like this:

case Checkout.execute(%{params: params}, options) do
  {:ok, %{order: order}} ->
    conn
    |> put_flash(:info, "Order completed!")
    |> redirect(to: Routes.order_path(conn, order))

  {:error, error}
    conn
    |> put_flash(:error, parse_error(error_description))
    |> render("checkout.html")
end

While writing this post, I created ex_pipeline, an open source library that allows developers to create pipelines while enforcing code style easily.

All code examples in this post are simplified versions of what exists in that project.

Chain of Transformations

One of my favorite hobbies is woodworking — I'm not good at it, by all means, but it's something that I do enjoy. Before starting a new project, I always write down all the materials and tools I'll need, as well as the steps I'll follow. That's how I go from pieces of wood to a finished table, from an initial state to a finished state.

Let's think about how a table is made: you get some raw materials — wood, nails, glue — and a few tools. Then you execute a set of actions to modify the shape and appearance of the materials — cutting, assembling, polishing, and painting.

You apply a chain of transformations on an initial state (raw materials) until you get the desired state (the table).

We already talked about the emerging pattern of a pipeline. That's what we are going to explore over the next paragraphs.

Writing Expressions as Reducers in Elixir

Reducers are one of the key concepts of functional programming. They are very simple to understand: you take a list of values, apply a function to each element, and accumulate the results.

In Elixir, we can work with reducers using the Enum.reduce/3
function and its variants — Enum.reduce/2 and Enum.reduce_while/3.

For example, we can use a reducer to filter all even numbers from a list:

numbers = [1, 2, 3, 4, 5, 6]
starting_value = []

Enum.reduce(numbers, starting_value, fn current_number, filtered_values ->
  if rem(current_number, 2) == 0 do   # is the current number even?
    [number | filtered_values]        # if so, add it to the accumulator
  else                                # otherwise...
    filtered_values                   # just ignore it and return the accumulator
  end
end)

This code will return the list [2, 4, 6].

To execute different actions on a value, we can transform the list of values into a list of functions. Then we pass the value as the initial value for the accumulator:

transformations = [
  fn x -> x + 1 end, # add 1
  fn x -> x * 2 end  # multiply by two
]
starting_value = 10

Enum.reduce(transformations, starting_value, fn transformation, current_value ->
  apply(transformation, [current_value])
end)

This looks like a complicated way to write (10 + 1) * 2, right? But by expressing each transformation as a function, we can attach an arbitrary number of transformations. They all must accept the same number of parameters and return the next updated state. In other words, these functions must look the same and be small.

In the end, we want to write the previous example as something like this:

defmodule MyModule do
  use Pipeline

  def sum_step(value, _), do: {:ok, value + 1}
  def double_step(value, _), do: {:ok, value * 2}
end

In the next part, we can do just that using a bit of magic from macros!

Sewing Reducers Into Features with Macros

Now that we know how to use reducers let's use them to write pipelines.

The goal is to make the process of writing, reading, and maintaining pipelines as easy as possible. So, instead of adding a DSL or using complex configs, we just want to write our functions using the same pattern of name and arity. We can then automate the process of building the pipeline by detecting which functions follow this pattern.

We create a new pipeline by writing code on the same pattern.

The State: Track What's Happening in Elixir

Before we dive into macros, let's take a step back and talk a little bit about the state of the pipeline.

When executing a pipeline, we need to know a few things: the initial state, the current state, if it's still valid, and if there are any errors. All this information will help us debug and troubleshoot any problems we might find when writing our code.

Also, by abstracting state updates in this module, we can enforce a very important rule on reducers: the return value must be an ok/error tuple.

The starting point of a module that manages state should be something like this:

defmodule Pipeline.State do
  defstruct [:initial_value, :value, :valid?, :errors]

  def new(value) do
    %Pipeline.State{valid?: true, initial_value: value, value: value, errors: []}
  end

  def update(state, {module, function_name} = _function_definition, options \\ []) do
    case apply(module, function_name, [state.value, options]) do
      {:ok, value} ->
        %Pipeline.State{state | value: value}

      {:error, error_description} ->
        %Pipeline.State{state | valid?: false, errors: [error_description | state.errors]}

      bad_return ->
        raise "Unexpected return value for pipeline step: #{inspect(bad_return)}"
    end
  end
end

The new/1 function will create a new valid and clean %Pipeline.State{} struct based on the initial value we give to the pipeline executor.

The update/3 function will update or invalidate the given state by calling function with the state's value and the given options.

If the given function returns an {:ok, <updated_value>}, then the state's value is updated, and we are good to go. However, if the function returns an {:error, <error>} value, the state is invalidated. The returned error is added to the list of errors in the state. If the function returns anything that's different from an ok/error tuple, it will raise an exception.

See the full implementation of this module.

Pipeline Steps and Hooks

Each function that transforms the state will be called a step. Here are some guidelines to follow:

• A step is a function whose name ends with _step and accepts two parameters.
• Steps execute in the order they are declared in their module.
• If one step fails, then all remaining steps are ignored.

Sometimes, we still want to execute code, even during failures — write a log, update a trace, publish a message, etc. We have a special type of step for these cases: a hook. Let's list the guidelines for hooks:

• A hook is a function whose name ends with _hook and accepts two parameters.
• Hooks execute in the order they are declared, just like steps.
• They are always executed at the end of the pipeline, after all steps execute.

Good! But how do we detect that steps and hooks exist? The answer is
metaprogramming! Luckily for us, Elixir has powerful metaprogramming tools to make all this possible.

Setting Up the Required Macros in Elixir

We'll need two things: macros and compile callbacks to read information about the functions of a module, and store what a step and a hook are.

Starting with the base module Pipeline, we want to use Pipeline in other modules. For that to happen, we declare a special macro, __using__/1. This macro is called with the keyword use in Elixir.

In this macro, we add a compile hook to the target module so that we can inject code into it. The @before_compile hook does exactly that by calling the given function when a compiler is about to generate the underlying bytecode. It accepts either a module or a tuple with a module and function name. We'll go with the second option to make things simpler.

defmodule Pipeline do
  defmacro __using__(_options) do
    quote do
      # Calls Pipeline.build_pipeline/1 before generating the final bytecode, allowing us to analyze, generate and
      # inject code whenever we call `use Pipeline`.
      @before_compile {Pipeline, :build_pipeline}
    end
  end

  defmacro build_pipeline(env) do
    # inject code!
  end
end

Anything called inside the quote block will be injected into the caller module — including the result of the @before_compile hook.

Check out the official docs for more information about quote and unquote blocks.

Now, whenever we want to create a pipeline, we call use:

defmodule MyPipeline do
  use Pipeline # <- code will be injected!
end

Detecting Steps and Hooks

We have everything in place to inject code, but we still need to generate the code that will be injected. Let's expand the build_pipeline/1 macro:

defmacro build_pipeline(env) do
  # Fetch all public functions from the caller module
  definitions = Module.definitions_in(env.module, :def)

  # Filter functions that are steps and hooks
  steps = filter_functions(env.module, definitions, "_step", 2)
  hooks = filter_functions(env.module, definitions, "_hook", 2)

  # Generate code on the caller module
  quote do
    # returns all steps and hooks
    def __pipeline__, do: {unquote(steps), unquote(hooks)}

    # Syntatic sugar so we can execute directly from the module that defines it
    def execute(value, options \\ []), do: Pipeline.execute(__MODULE__, value, options)
  end

  # Filter functions from the given module based on their suffix and arity
  defp filter_functions(module, definitions, suffix, expected_arity) do
    functions =
      Enum.reduce(definitions, [], fn {function, arity}, acc ->
        # Detect if the function name ends with the desired suffix: _step or _hook
        valid_name? =
          function
          |> Atom.to_string()
          |> String.ends_with?(suffix)

        # Detect if the function arity matches the expected arity
        has_expected_args? = arity == expected_arity

        cond do
          valid_name? and has_expected_args? ->
            # In this case, the function does end with the desired suffix and has the correct arity.
            # We include it in the accumulator along with the line where it was declared so we can
            # order them by their position in their module
            {_, _, [line: line], _} = Module.get_definition(module, {function, arity})
            [{module, function, line} | acc]

          valid_name? ->
            # If the function has the desired suffix but the arity is not correct, we raise this error
            # because we don't want people trying to figure out why their steps or hooks are not
            # being executed.
            raise(PipelineError, "Function #{function} does not accept #{expected_arity} parameters.")

          true ->
            # If the function doesn't match our filters, it's not a part of the pipeline and can be
            # just ignored
            acc
        end
      end)

    # At this point we have filtered the functions that match the filter criteria but they
    # are not in order. We use the line number information to sort them and then drop it
    # once we don't need it anymore.
    functions
    |> Enum.sort(fn {_, _, a}, {_, _, b} -> a <= b end) # order by line number
    |> Enum.map(fn {m, f, _l} -> {m, f} end)            # drop line number
  end
end

This macro receives the compiler env, which has lots of context information. Here, the important bit is the caller module, accessible via the env.module attribute.

Then we use the module.definitions_in/2 special function to get a list of all functions declared on the module env.module with the def keyword.

We call the Pipeline.filter_functions/4 function to filter these definitions by their suffix and arity, essentially detecting our steps and hooks! The Pipeline.filter_functions/4 function is kind of big, so I've added comments to help you navigate through it.

As said before, anything inside the quote block is injected directly into the caller module. So whenever we call use Pipeline, the module will have two extra functions: __pipeline__/0, and execute/2.

The Pipeline module uses the first function to execute our custom pipeline, while the execute/2 function is just a convenience function that will execute Pipeline.execute/3. It allows us to execute our pipeline by calling MyPipeline.execute(value, options).

Speaking of Pipeline.execute/3, it's time to define it. This function is the core of this engine, and it's responsible for actually calling a reducer that will power our pipeline engine:

defmodule Pipeline do
  # ...

  def execute(module, value, options) do
    # Build a new initial state of the pipeline
    initial_state = State.new(value)

    # Fetch steps and hooks from the module
    {steps, hooks} = apply(module, :__pipeline__, [])

    # Run each step from the pipeline and build the final state
    final_state =
      Enum.reduce(steps, initial_state, fn reducer, current_state ->
        State.update(current_state, reducer, options)
      end)

    # Run each hook with the final state
    Enum.each(hooks, fn {module, function} ->
      apply(module, function, [final_state, options])
    end)

    # Transform the state value into an ok/error tuple
    case final_state do
      %State{valid?: true, value: value} ->
        {:ok, value}

      %State{errors: errors} ->
        {:error, errors}
    end
  end
end

Check out the complete version of this module here.

Note: In the complete version of this module, there is also another kind of hook, called an async hook. Async hooks work just like the hooks presented here, but are executed in parallel rather than sequentially. They will not be covered in this post, since they are essentially a hook executed with a Task.

Building and Executing a Pipeline

By simply calling use Pipeline and writing function names with the _step or _hook suffix, we are now able to build our pipelines. Let's rewrite our last example from the first part of the post with this new mechanic:

defmodule Checkout do
  use Pipeline

  def fetch_payment_information_step(value, options) do
    # ...
  end

  def fetch_user_step(value, options) do
    # ...
  end

  def fetch_address_step(value, options) do
    # ...
  end

  def create_order_step(value, options) do
    # ...
  end

  def report_checkout_attempt_hook(state, options) do
    # ...
  end
end

We can execute this new pipeline like this:

case Checkout.execute(%{params: params}, options) do
  {:ok, %{order: order}} ->
    conn
    |> put_flash(:info, "Order completed!")
    |> redirect(to: Routes.order_path(conn, order))

  {:error, error}
    conn
    |> put_flash(:error, parse_error(error_description))
    |> render("checkout.html")
end

No more with blocks, and no need to manually handle errors. The controller is now focused on handling the HTTP layer, and the checkout feature has an official place to live. Adding a new step to the checkout process just requires that we write a function in the right place!

Wrap Up

In this post, we learned how to express a set of transformations as a reducer. Then, using the power of macros, we automatically created pipelines by detecting the signature of functions at compile time.

The features that use pipeline mechanics are explicit and contained — we know exactly what to expect and where to go when modifying them. We expect all steps to look and behave in the same way. The basics of error handling are already in place, you just need to decide what to do. These pipelines are, indeed, predictable.

I hope you've found this two-part series about keeping your Elixir code maintainable helpful. Until next time, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!