Untestable: testing compile-safe strategies via `dynamic`

If we fix a bug by making a C# program more type safe, how do we test this?

Say for example you arrange for invalid code to not compile anymore?

The case

Active Logic uses a modified Kleene logic to implement Behavior Trees using the short-circuiting operators && and ||. In addition of a three-valued type, we have restricted types to represent ahead-of-runtime knowledge. Anyway this is handy because it avoids logical errors (we love uncertainty, in moderation).

So here's an example of an operation that we're disallowing:

impending x;
status y;
var z = x && y;

impending represents a task that's either running or failing; whereas status (typical of BT) is either running, failing or done. In the above example, y (which normally is an expression, not a pre-assigned value) would never evaluate.

So this is something we can enforce at compile time, and the question then becomes: how do you put this under test.

Initially, our test looked like this:

[Test] public void Impending_AND_Status(){
    impending x = impending_fail;
    status    y = done;
    var z = x && y;  // CS217
}

With our implementation this raises CS217 at compile-time, invalidating the (impending) && (status) combination.

Expected behavior; also, who broke my test suite?

The dynamic keyword lets you disable compile checks:

[Test] public void Impending_AND_Status(){
    dynamic x = impending.fail, y = status.done;
    var z = x && y;
    Print($"{x} && {y} => {z}");
}

...and you'd expect a runtime error to assert against. However (in our case) things get a little more complicated because the above DOES run and produce an output.

The compiler sees an invalid logical AND, but the runtime takes a more incremental approach:

1) Check the left hand for "falsehood". If the left hand is false return the left hand.
2) Type-check the right hand. If there is an applicable & operation of the form lh & rh, evaluate the right hand.
3) Invoke the (user defined) & operator on the resulting operands.

With this in mind, I then rewrote the false operator for the impending type:

public static bool operator false(impending s)
=> throw new InvOp("Cannot test falsehood (impending)");

NOTE: In C#, true and false work in pairs. You can't implement one and not another. pending should just never allow &&. But this, ultimately, is not something we can enforce at compile time.

And our test then looks like this:

[Test] public void Impending_AND_Status(){
    dynamic x = impending_fail, y = done;
    Assert.Throws<InvOp> ( () => z = x && y  );
}

Downside of course, the compiler and runtime aren't checking the same thing. Strictly we'd have to forgo NUnit, and write a script that:

1) Runs a build on invalid code
2) Verifies every error issued by the compiler

This approach, however, is heavy-handed, so I decided to go ahead with runtime checking and here's the final version of the test, as it will look in the next commit:

// o(x, y) // assert x equals y
// s(x)    // new status from an int
// i(x)    // new 'impending' value from int
[Test] public void Impending_x_Status([Range(-1, 0)] int x,
                                      [Range(-1, 1)] int y){
    AND_CS217_InvOp(i(x), s(y));
    o( i(x) || s(y), s(x) || s(y) );
}

AND_CS217_InvOp encapsulates the NUnit call (which is a bit long winded when you have (7 x 7 x 2) cases to cover). The compile error is still no more than a label, still useful for documenting our APIs.

Cooler talk

With runtime testing, compile-safe techniques are in a blind spot. This doesn't sit too well with me because testing is supposed to guide you towards better code. Is compile time safety not a good thing, then?

When you fix a bug or add a feature by tweaking type safety, if you don't have a test you also can't ensure your bug fix/feature won't be removed accidentally. You can roll your own tools, and dynamic is a poor man's version of doing just that. Oh, and it helps understanding how the C# runtime even works.

Photo by Ali Sarvari on Unsplash