Taking Go Generics for a Spin

Generics are a part of most major programming language as a way to use types as parameters to function, classes, structures and so on. Historically, GoLang had been known for being a popular and newer programming language, without the added complexity of generics, which was commonly regarded as a bonus in some circles. Nevertheless, this all changed as 1.18 rolled out with a shiny new feature: generics. They're so new that even the standard libraries aren't fully migrated to generics yet. I thought I'd take generics out for a spin and tackle a problem I've not yet seen addressed: loading environment variables and processing them to their desired types in an eloquent way. Disclaimer: this was my first experiment with generics in Go, so take it with a grain of salt. Here goes.

The Syntax

First of all, languages like Java have generic type arguments in their DNA, so much so that mostly anything can be extended with generic type arguments, however in Go, the approach is somewhat different. The first limitation is that generics can not be used on methods, only functions. Some might see this as a non-issue, given that methods are just syntactic sugar for defining the same function with the "instance" as the first argument and then calling it directly with the instance passed in as a parameter. A bit more verbose, but gets the job done. We won't be doing that today, though. You can add generic type parameters to functions and make types generic. This system isa bit limited, but it's actually perfectly in line with the philosophy of Go: the less stuff you can do, the fewer the ways you can invent to solve problems and thus your code will be closer to everyone else's. In today's post I'm exploring adding generic type arguments to functions.

To do so, I can use the following syntax:

// here I pass in T as a generic argument
func At[T any](arg []T, i int) T {
  return arg[i]
}

Couple of things to note. Whenever you pass in a type parameter, you must also define what it conforms to. To make it a bit easier, the GoLang team mapped our favourite type interface{} to the keyword any, which will conform to anything you give it. You can use this mechanism to make sure the passed in argument conforms to a certain interface, such as comparable if you want to sort, order or filter things.

You can also define the type of inputs and pass them into generic types or turn them into slices.

The function can take regular parameters too, but honestly, what would be the point if it couldn't.

You can also tell the function that the return type has something to do with the generic argument too.

Lazy Env

My library will be called lazyenv. It's a small library with a single purpose: allowing the user to get environment variables in a lazy way - caching those on the first instance of retrieving the variable and also converting them to the desired type in an eloquent way.

Let's look at an example of what I'm trying to achieve. Say we have an environment variable as BUSLINES=7,8,108,133,907,908,932,956,973,990 and in my program I want to have a list of these bus lines. Now I know that all buses have a positive number between 0-999, which means we can use an unsigned integer. 8 bits will not cover our full range, but 16 bits will. In pseudo code, I want my library's interface to look like this:

buslines = get "BUSLINES" as uint16

Designing an Interface

As I've mentioned before I want to design an eloquent interface for users to interact with my library. An eloquent interface is one that reads as close to human language as possible, but without abstracting function and variable names too much that in the process they become completely disconnected from their actual purpose. Let's see an example of an eloquent interface for Go, based on the above design.

buslines := lazyenv.Get("BUSLINES").Uint16Slice()

To implement something like this, we'd do the following

package lazyenv

import (
    "os"
    "strconv"
    "strings"
)

type env struct {
    value string
}

func Get(key string) env {
    value := os.Getenv(key)
    return env{value}
}

func (e env) Uint16Slice() []uint16 {
    vals := strings.Split(e.value, ",")
    var ret []uint16
    for _, v := range vals {
        i, _ := strconv.ParseUint(v, 10, 16)
        ret = append(ret, uint16(i))
    }
    return ret
}


var buslines = Get("BUS_LINES").Uint16Slice()

But wait, where are the generics?

Don't worry, we'll get to that soon. A small hint: we might not end up using this API.

Next thing I want to add is the ability to define a default value that should be returned, if the environment variable is empty.

The Builder Pattern

The pattern we're using here is based on the builder pattern. In the builder pattern we use a base object, class or struct to accumulate the information we're building. We're using methods on the instance of the builder to "build" values. When we're done building, we finalise the result by calling a designated method or in some cases, multiple methods can yield a result, like in my example, where we could finalise the builder to a .String(), .Uint16Slice() or .Interface() among others.

type env struct {
    value string
    defaultValue interface{}
}

func (e env) Default(value interface{}) env {
    e.defaultValue = value
    return e
}

Let's add some generics:

type env[T any] struct {
    value string
    defaultValue T
}

The env struct now supports a generic argument. This will light up our code like a Christmas tree. We need to supply this generic argument everywhere to satisfy the compiler.

func Get[T any](key string) env[T] {
    value := os.Getenv(key)
    var defaultValue T
    return env[T]{value, defaultValue}
}

Our builders methods now look like this:

func (e env[T]) Default(value T) env[T] {
    e.defaultValue = value
    return e
}

func (e env[T]) Uint16Slice() []uint16 {
    vals := strings.Split(e.value, ",")
    var ret []uint16
    for _, v := range vals {
        i, _ := strconv.ParseUint(v, 10, 16)
        ret = append(ret, uint16(i))
    }
    return ret
}

The compiler can technically infer this argument, however in this specific situation it isn't possible because we're moving backwards. We're only instructing with actual values what the type should be once the struct is already built.

What we do get, however is type safety. We can't supply a wrong default value, because our program won't compile.

Here's what our code to get this variable becomes now:

var buslines = Get[[]uint16]("BUS_LINES").Default([]uint16{}).Uint16Slice()

What's weird about this code now, is that we have to declare our type twice.

Another problem is the logic to make default values work will now present us with a little problem.

There is actually nothing stopping you from providing a []uint16 as the default value but asking for a .String() to be returned. Our default value and struct are already typed at this point, but sadly there's no way for us to grab an initialiser for that type, because type information isn't dynamic at runtime, it is - again, just syntactic sugar, that let's you write a function with the same signature once, but in reality will work the same way as though you typed that function in place. In other words, the following just isn't possible:

func AccessType[T any](t T) {
    switch T {
    case uint16:
        fmt.Println("uint16")
    case string:
        fmt.Println("string")
    }
}

This will bring us to the problem at hand:

My other gripe with this whole approach is extensibility. In Swift, it is possible to extend any interface and it will all just magically end up being wired together. Swift is a much more complex language, however, and Go does not allow you to do such a thing, so using a builder pattern will mean that ultimately this library will ultimately be limited by the mappings I add to it as finaliser methods. One can't come in and provide third-party mappings to different types.

At this point, I know I'll need more generic functions or methods, but I also want to pass in the type at the earliest possible time. Let's drop the builder pattern for now and use function composition instead.

Function Composition

With the function composition pattern, we provide a single point of entry, much like with the builder pattern, however we define additional behaviour by introducing functions as parameters. Here, I'll need a mapper and a default value getter function. I elected to use a function to return the default value rather than just passing it in for 2 reasons:

1. This makes it more lazy - meaning that if we don't end up needing a default value, the program won't have to initialise it. The tradeoff is that if our default value was just a primitive value or something simple and not resource intensive, then we'd end up with more overhead, however if, for example, we want to fetch the values from an API if the environment doesn't provide them, we can do that in an eloquent way, without sacrificing performance.
2. This allows for function composition to control the behaviour of the fallback logic. E.g.: this lets us make it so that no default value is returned, but the program returns an error instead or we can even panic right away.

So why not just use parameters to fine-tune behaviour?

If you're already sold on function composition, feel free to skip this paragraph, but for those of your skeptics out there, I present:

// a simple interface
type get [T any]func (key string) (T, error)

Now let's use parameters to define the behaviour of this function. Should we return the error if no value? Should we just not do anything? Should we panic?

type get [T any]func (key string, returnError bool, panicOnMissing bool) T

Where does the default value go?

type get [T any]func (key string, returnError bool, panicOnMissing bool, defaultValue T) T

So now our invocation looks like this:

Get[string]("test", true, false, "")

Our implementation is arguably also a mess and there is no constraint stopping us from passing in true for both behaviours and also adding a default value, in which case, which one takes precedence?

var val string
if returnError && panicOnMissing {
    panic(errors.New("returnError and panicOnMissing cannot both be true"))
}
if returnError {
    return val, errors.New("")
}
if panicOnMissing {
    panic("")
}
return val, nil

When you look at the above line, do you know what each parameter does? Even I don't remember and I wrote it not a minute ago. So what would this look like with function composition?

type get [T any]func (key string, onMissing func(string) (T, error)) (T, error)

Let's take a stab at implementing panic on missing value:

func OrPanic[T any](key string) (T, error) {
    panic(fmt.Errorf("%s is not set", key))
}

To use it, one can pass it into our getter, like so:

Get("EXAMPLE", OrPanic[string])

Note, we instantiate OrPanic here with the string type, which also allows Get to infer the type or T to be. This is mandatory, otherwise Go wouldn't know what type it has to apply during build.

Let's take this a step further and implement returning an error if our value is missing:

func OrError[T any](key string) (T, error) {
    var value T
    return value, fmt.Errorf("failed to get %s", key)
}

We can only use one or the other, so that takes care of transparently deterministic behaviour:

Get("EXAMPLE", OrError[string])

Finally, we can provide a default value, like so:

func OrReturn[T any] (defaultValue T) func(key string) (T, error) {
    return func(key string) (T, error) {
        return defaultValue, nil
    }
}

Notice, how we're using a factory to create a function that matches the signature that we need for the parameter that we pass to Get. Here's what the usage looks like:

Get("BUSLINES", OrReturn([]uint16{}))

One great thing to note here, is that because we're actually passing in a parameter here that we annotated with the generic type argument T, we do not have to also instantiate the type parameter, because of inference.

Let's map these originally string values to something more useful. Let's try an int first. Those of you familiar with the strconv library will notice that it actually comes with a function that we can already use as is, because it matches our signature of the mapper. I'll let you take a guess what it is. Leave a comment down below if you got it correct!

type Mapper[T any] func(value string) (T, error)

Our Get function now becomes

func Get[T any](key string, getDefaultValue GetDefaultValue[T], mapper Mapper[T]) (T, error)

We can use it as such:

func String(value string) (string, error) {
    return value, nil
}

var s, err = Get("EXAMPLE", Optional[string], String)

strconv.Atoi also adheres to this interface, so it's possible to map to an int right away:

var i, err = Get("NUMBER", Optional[int], strconv.Atoi)

But we can make it more fluent and just keep a function at hand for this specific task too:

func Int(value string) (int, error) {
    return strconv.Atoi(value)
}

Now we need to find a way to get our slice of numbers in our list separated by ,. We'll take advantage of composition and generic arguments yet again, to achieve such feat.

I'm going to design a function, that takes a Mapper as an argument and returns a slice of the type of whatever the Mapper maps to.

func SliceOf[T any](mapper Mapper[T]) Mapper[[]T]

I can then take this function and pass it into my Get to maintain the eloquence (is that even a word?) of this library, while adding new functionality. Keep in mind, I'm now just adding on top of my already existing "platform", which is the extensible API I've designed earlier. One could theoretically implement any mapper or mapper factory based on their specific needs.

The final example for my use case is:

buslines := Get("BUSLINES", Required[[]int], SliceOf(Int))

Ambiguity

We have one more issue we need to fix before we can move on to implementing this library. Currently our GetDefaultValue and Mapper types are in fact the same. This means that Go will let you pass in a Mapper for GetDefaultValue and vica versa. A language feature that would resolve this problem is lexical type assertion, where one can define a given variable, or function as a certain type and that is enforced via the type system, even if the two types in question conform to each other. Go's type system is extremely flexible and as such the mechanism I described doesn't exist within it. Instead we can just change the signatures of the function, which is slightly inconvenient, however it will let us differentiate between the two different types of functions we export and use.

type GetDefaultValueParams struct {
    Key string
}

type GetDefaultValue[T any] func(params GetDefaultValueParams) (T, error)

Aside about tests

I've made this change after finishing the implementation itself and with 100% test coverage. After updating the implementation to match the new logic, I didn't have to touch any of my tests and they still passed, which is a sign of a robust test suite - it lets you refactor freely, as long as you retain all of the original functionality.

Conclusion

I am personally a huge fan of generics. The generic system in Go is a controversial one, because it does remove from the simplicity of the language that it was designed for. It also isn't as flexible and "smart" as other ones. Java's generics system is more robust, while the one in TypeScript is a lot more proactive with inference, which leaves you typing less code. The long-term effect of having generics as part of the language is going to be a positive one, especially as soon as the standard library gets reworked to make use of generics.

If you want to see the implementation of this library or just want to use it in your project, I've put all of the source code on GitHub.