DEV Community

ilya mozerov
ilya mozerov

Posted on

A test that passed for the wrong reason

A test that checks an exit code can pass for two different reasons. If two different causes end with the same code, the test cannot tell them apart, and it will keep passing after the thing it was written to guard has stopped working.

We found this in our own privacy scan this month.

What the scan promises

The scan reads every article we have published and every draft, checks them against a list of private terms, and publishes one verdict: GREEN, RED, or UNKNOWN. The contract has one rule that matters here. If the scan cannot publish its verdict, it must not stay silent. It returns exit code 3, which means UNKNOWN, and a stale GREEN is never allowed to stand in for it.

Two tests guard the failure path. When the publish step fails, a RED must stay RED. It must not quietly become UNKNOWN, and it must not become GREEN.

The tests that could not fail

Both tests called the publish function without naming which command-line tool to publish with. The function then fell back to a default path. On the machine where that file was absent, the code took a branch that said "no tool available" and exited with code 3.

A failed publish also exits with code 3.

So the tests asserted exit code 3, and they passed. They would have passed just as well if the publish had never been attempted at all. The branch they claimed to test was never reached. Nothing in the test could see the difference between "the publish was tried and failed" and "the publish was never tried".

A reviewer refused the change as suspicious. The reviewer did not doubt the scan's logic. The reviewer pointed out that the tests could not distinguish the branch they named.

The repair, in two steps

First, the tests were given a real stand-in tool, so the publish attempt actually reached the test's fake runner. That alone was not enough. The tests still checked only the exit code, and the two causes still shared it.

Second, the tests asserted the branch itself. The runner must have been called, and it must have been called with the tool the case named. A third case states the ambiguity outright. A missing tool and a failed publish both exit with 3, and the two differ in exactly one observable thing: whether the publish was attempted. That third case exists so the assertions cannot be waved away as decoration.

Breaking it on purpose

A test that has never failed has not been shown to work. We broke the code in two controlled ways and restored each one after the test had reacted.

When the RED-stays-RED rule was reverted to UNKNOWN, the test failed with exit code 3 where it expected 2. That is the right failure, and the old tests would have missed it.

When the publish was made to return before reaching the runner, three cases failed with the message that the publish was never attempted. The old tests passed through that break without a word.

What we took from it

A test on an exit code checks the code and nothing else. When a failure and a missing dependency can share a code, assert the one observable that differs between them, and then break the code on purpose to watch the test fail.

The fixed tests went out through the normal delivery path. The scan runs hourly as an installed timer, and the last verdict we checked read GREEN.

We think the lesson is bigger than one scan. Most green checks in a system are green for some reason. The useful question is which other reason would also make them green.

Top comments (0)