Converting a nilable generic argument to its non-nil version

I have struggled a great deal in the past with making generic code that provides functionality to transform a generic argument from a nilable version to a non-nilable version. And I just learned something pretty cool today.

Currently, I am working on a parsing and validation library and want to provide the ability to type-safely move from a nilable string input, to a non-nilable output without directly tying the library to only working with strings are optional inputs.

Here's some simple example code

class Parser(T)
  private getter input : T

  def initialize(@input)
  end

  def not_nil
    if temp = input
      return temp
    else
      raise "input was supposed to be present, but was nil"
    end
  end
end

input = "raw_input".as(String?)
Parser.new(input).not_nil #=> "raw_input" : String

input = nil.as(String?)
Parser.new(input).not_nil #=> runtime exception!

I'm hoping that code example is pretty straightforward. It shows a very basic Parser class that has one method called Parser#not_nil. It returns the non-nilable version of whatever you give it, otherwise it raises an exception if the input actually is nil. In the example, I gave it a nilable string and it returns a non-nilable string. If you are confused as to why I use a temp variable, read more on that here.

One thing to notice about this code example, is that there is no way to express the return type of Parser#not_nil so I am forced to let Crystal infer it. It works as expected but I am not overly fond of leaving off return types. It could easily infer something I didn't expect because there is a bug in the code and it also leaves the documentation lacking.

This code is not very good, though. As I was saying, I'm working on a parsing and validation library so instead of an exception, I'd prefer a way to express this so that it results in a value that I can return to the user.

If your familiar with functional languages (or Rust), I'm talking about implementing a Result type of sorts. A Result type is a data structure that can only be represented as one of two options: an Ok or an Err (or Error to be more explicit). This is implemented generically to where the Result wraps the value contained in either the Ok or Err.
For my purposes, I want the Ok to wrap any value T and the Err to always wrap a String. With Crystal generics you'd write the Result as Result(String, T) where the first item in the parens is the Err type and the second item is the Ok type. To not continue the article with two generics, and because it more closely matches my actual implementation, I will use a Validated(T) type that is essentially equal to a Result(String, T) and has two implementing types of Valid and Invalid.

Here's a real basic implementation of the Validated class.

abstract class Validated(T)
  class Valid(T) < Validated(T)
    getter value : T
    def initialize(@value)
    end
  end

  class Invalid(T) < Validated(T)
    getter value : String
    def initialize(@value)
    end
  end
end

There's very little code here, but it's the basic implementation that matter. I know it looks weird for Validated::Invalid to also have the generic type on it since it only deals with string values (the error messages), but it relates to the code that is missing and it would be more difficult to illustrate what I'm talking about without it. In particular, the code that is missing, involves converting Valid to Invalid and vice-versa. You couldn't do that without keeping up with the valid type on the invalid classes.

FINALLY, let's get to the problem code so that I can show off what I learned today.

class Parser(T)
  private getter input : T

  def initialize(@input)
  end

  def not_nil
    if temp = input
      Validated::Valid.new(temp)
    else
      Validated::Invalid.new("Expected input to be present, but was nil.")
    end
  end
end

input = "raw_input".as(String?)
Parser.new(input).not_nil.value #=> compilation error!

If you actually try to compile this, it fails. It fails because the compiler has no idea what the generic type is supposed to be for the Invalid class. The argument that we pass to create it, is the validation error message after all and we want it to be the not-nil type. We have finally reached the problem that I have run into multiple times and next step has time and again been to open up Discord, go to the Crystal channel, complain, then go do something else while wondering in the back of my mind whether or not I should give up on Crystal and use another language because I figured it was unsolvable. But I decided to do something differently today. I decided to look into where I knew Crystal did something similarly. Specifically, I know that in Crystal you can call [1, 2, nil, 3].compact and get back the array [1, 2, 3] with a type of Array(Int32). The compiler assures that there can't be nil with the type signature. If the core language is able to express a change from T? to T, I should be able to as well. So I opened up the core codebase and found that Array#compact delegates to Enumerable#compact_map. Here's the code for that method

def compact_map(& : T -> _)
  ary = [] of typeof((yield Enumerable.element_type(self)).not_nil!)
  each do |e|
    v = yield e
    unless v.nil?
      ary << v
    end
  end
  ary
end

This took me a while to understand. It's calling this Enumerable.element_type function and yielding with it and I still don't understand that part, but the important piece is that it's calling typeof((...).not_nil!).

🤯🤯🤯🤯

I had no idea that you could call not_nil! inside typeof without raising an error if the thing you are calling it on is actually nil.

x = nil.as(String?)
typeof(x.not_nil!) #=> String : Class

🤯🤯🤯🤯

So Enumerable#compact_map is able to let the compiler infer the return type by creating the return array with the non-nilable type.

Now for the grand reveal of how I was able to use this in my own code.

class Parser(T)
  private getter input : T

  def initialize(@input)
  end

  def not_nil
    if temp = input
      Validated::Valid.new(temp)
    else
      Validated::Invalid(typeof(input.not_nil!)).new("Expected input to be present, but was nil.")
    end
  end
end

input = "raw_input".as(String?)
Parser.new(input).not_nil.value #=> "raw_input" : String

input = nil.as(String?)
Parser.new(input).not_nil.value #=> "Expected input to be present, but was nil." : String

Hey!! Look at that!! It works!!

So while I use the temp to allow the if-block to narrow the type down, I need to avoid doing that with the typeof((...).not_nil!) trick so that it remains the original type. The compiler now understands that, while it doesn't have a reference to the not-nil value, Invalid has the same generic type as Valid.

I love this, and I'm going to run with it in my code that does a whole lot more that's outside the scope of this post but I will hopefully have it ready soon to share. I hope this will be helpful to someone else and that your mind is as blown as mine is.

By the way, I have noticed the core team refers to this hack in generic terms as T -> T! and would love for this to be implemented in Crystal without the hacks.