The Parable of the Unit Tests

Somewhere, some day, a developer is writing a simple helper function to calculate the (integer) mid-point between two integers.

object Util {
  def average(a: Int, b: Int): Int = (a + b) / 2
}

They know that if there is no unit test their code-reviewer will call them on it, so they dutifully write one.

class UtilSpec extends FreeSpec with Matchers {
  import Util._

  "calculating an average" - {
    "comes up with the right result" in {
      average(2, 4) shouldBe 3
    }
  }
}

It passes.

[info] UtilSpec:
[info] calculating an average
[info] - comes up with the right result
[info] All tests passed.

The developer has achieved 100% test coverage! But they still feel guilty. This is the worst kind of “happy path” testing ever! Surely they need more tests to demonstrate the function works with different kinds of input.

So our developer assuages their guilty conscience with more tests.

"calculating an average" - {
  "comes up with the right result" in {
    average(2, 4) shouldBe 3
    average(4, 2) shouldBe 3
  }

  "works if either of the numbers is zero" in {
    average(0, 0) shouldBe 0
    average(0, 4) shouldBe 2
    average(4, 0) shouldBe 2
  }

  "works for positive and negative numbers" in {
    average(-2, -4) shouldBe -3
    average(-4, 2) shouldBe -1
    average(2, -2) shouldBe 0
  }

  "rounds in the expected direction" in {
     average(2, 3) shouldBe 2
     average(-2, -3) shouldBe -2
  }
}

And they pass too.

[info] UtilSpec:
[info] calculating an average
[info] - comes up with the right result
[info] - works if one of the numbers is zero
[info] - works for positive and negative numbers
[info] - rounds in the expected direction
[info] All tests passed.

“There, that's better!” Four tests, no fewer than ten separate assertions showing the code works in all sorts of different situations. This should make the reviewer happy.

Another developer passing by looks over the first developer's shoulder and asks if they might pair. After a short moment’s thought, the new pair of eyes suggests deleting all the tests the first developer wrote, replacing them with just one test:

class UtilSpec extends FreeSpec with PropertyChecks with Matchers {
  import Util._

  "calculating an average" - {
    "returns a result between the two operands" in {
      forAll { (x: Int, y: Int) ⇒
        average(x, y) should (be >= Math.min(x, y) and be <= Math.max(x, y))
      }
    }
  }
}

“Wait a minute”, says the first developer. “That’s not testing the right thing! There’s dozens of different functions I could write that satisfy that invariant but don’t calculate an average!”

”But I knew before I sat down”, says the second developer. “that you didn’t write any of those functions.”

[info] UtilSpec:
[info] calculating an average
[info] - returns a result between the two operands *** FAILED ***
[info]   TestFailedException was thrown during property evaluation.
[info]     Message: -1073741821 was not greater than or equal to 7
[info]     Location: (UtilSpec.scala:19)
[info]     Occurred when passed generated values (
[info]       arg0 = 7, // 8 shrinks
[info]       arg1 = 2147483647
[info]     )
[info] *** 1 TEST FAILED ***