Mutation testing in PHP

Recently I spent an afternoon experimenting with mutation testing in PHP. In this post I would like to share the background, the main idea of mutation testing, and the lessons I’ve learned from it.

Moxio Academy

At Moxio we regularly schedule a Moxio Academy session. That means we put our normal work aside for an afternoon to learn something new or experiment with the latest technologies. You can choose any topic, as long as it benefits your personal development and thus your role at the company. There are just a few rules:

• You have to decide on a topic and announce it in advance. This is meant to improve synergy within the team: maybe someone already knows a lot about the topic and has some resources to recommend. If multiple people want to learn about the same thing, maybe they can combine their efforts and learn together.
• You must create a plan for how to approach your topic. This does not have to be very extensive, just a rough sketch is enough. The idea is that this forces you to focus your learning efforts rather than just reading random resources without any sense of a general direction.
• You should work towards some kind of deliverable. Ideally this is something that can be shared with the rest of the team: a tool, wiki article, some proof-of-concept, blogpost or lunch-and-learn presentation. Again this helps to focus the process, to have some concrete end result, and to spread the acquired knowledge within the team.

We usually conclude the afternoon together to present our process and results, and (for those who want) have a pizza.

Mutation testing

At the latest Moxio Academy I wanted to experiment with mutation testing in PHP. I had already heard about mutation testing and Infection as a tool to do so a few times, but never got around to play with it hands-on. This seemed like a good opportunity.

In essence, mutation testing is a way of measuring the quality of a test suite. A tool generates a number of copies (mutants) of your source code under test, but modifies each of them in a small way. These small modifications are basically just errors that you as a programmer could have made, such as replacing a < with a <=. A high-quality testsuite would detect these modifications (kill the mutant) by failing one or more tests. In a suboptimal test suite it might happen that the tests remain green despite the modification to the source code. In such a case we speak about an escaped mutant. These present an opportunity to improve the test suite by adding a test that fails in the presence of the given modification. The Mutation Score Indicator (MSI, the percentage of mutants detected by the test set) provides a metric for the quality of the test suite.

My plan was to apply Infection to one of our internal libraries. I picked this library because of its high (line-based) code coverage, which would imply a high-quality test suite. Therefore I wondered what ‘leaks’ Infection could still find in it. The idea was to convert the escaped mutations found by Infection to new tests, working towards a PR to increase the MSI of the project as the main deliverable. This blog post wasn’t part of the original plan, but arose as an extra way of sharing some of the lessons learned along the way.

Lessons learned

Mutation testing is awesome!

My main takeaway was that mutation testing can really help you to improve your test suite. Even on a project with 96% line coverage, Infection found multiple scenarios that were not actually covered by the test suite.

One simplified example of this is the following. Suppose we have a function to generate a description for the amount of bottles of beer on the wall:

<?php
function describeBottles(int $amount = 99): string {
    return $amount . ' bottles of beer on the wall';
}

We could already have a test for this function like this:

<?php
class DescribeBottlesTest {
    public function testDescribesTheAmountOfBottlesOfBeer() {
        $this->assertSame('42 bottles of beer on the wall', describeBottles(42));
    }
}

At first sight it may seem that this fully tests the function, and indeed the function shows up with all lines covered in a code coverage report. However, our tests do not check the default value for the argument. This means that some behavior of our function, i.e. that by default it describes 99 bottles, is not verified. Infection can uncover this when it produces a mutation like:

12) /tmp/bottles.php:2    [M] DecrementInteger

--- Original
+++ New
@@ @@
<?php
- function describeBottles(int $amount = 99): string {
+ function describeBottles(int $amount = 98): string {
      return $amount . ' bottles of beer on the wall';
  }

Here the DecrementInteger mutator has decremented an integer literal occurring in the source code, an error we could have made ourselves if we hit the wrong key on our keyboard. This would currently go unnoticed, but we can fix that by adding a test like:

<?php
class DescribeBottlesTest {
    public function testDescribes99BottlesByDefault() {
        $this->assertSame('99 bottles of beer on the wall', describeBottles());
    }
}

Other untested aspects commonly found by Infection were exception messages and some uncovered paths through complex logic. I added tests or assertions for most of these. For the logic, this is also a sign that the (cyclomatic/N-path) complexity is too high. Those pieces of code should be refactored, but I scheduled that for later.

On the first Infection run, without any changes to the test suite, it produced a 89% MSI. This is already quite good, but with some additions to the test suite I managed to raise the MSI to 93%.

‘False positives’

Getting the MSI much higher proved to be difficult though. Sometimes escaped mutants had changes to parts of the code that are nonessential details. In our view, these details do not constitute relevant behavior and are not part of the ‘contract’ of that unit. Why are they in the code then? Well, sometimes they have to be due to syntactical constraints. Take for example the following piece of code that may throw an exception:

<?php
// ...
try {
    $database->executeSql("...");
} catch (DuplicateDatabaseKeyException $e) {
    throw new UserAlreadyExistsExeption("User $username already exists", 0, $e); 
}

As explained in a previous blog post, we think it is important that exceptions are thrown at the right level of abstraction:

Best practices for PHP exception handling

Arnout Boks ・ Jan 10 '19 ・ 9 min read

#php #exceptions #oop #softwaredesign

This requires catching and re-throwing exceptions like in the above code snippet. In that post, I also mentioned that we want to maintain the connection with the root cause by setting the $previous-parameter of the new exception. Due to PHP's syntax this requires one to also provide the $code-parameter, which we do not really use. We usually set it to 0 (the default), but honestly couldn't care less about its value.

Now the same DecrementInteger mutator could come along and produce this mutation:

53) /tmp/exception.php:289   [M] DecrementInteger

--- Original
+++ New
@@ @@
  } catch (DuplicateDatabaseKeyException $e) {
-     throw new UserAlreadyExistsExeption("User $username already exists", 0, $e); 
+     throw new UserAlreadyExistsExeption("User $username already exists", -1, $e); 
  }

This mutation will not get caught by our test suite (because we do not assert the code of produced exceptions), so the mutant will escape. In this case we do not care however. We don’t expect our test suite to detect this, as (to us) the mutated code is just as fine as the original. We could add assertions for the exception code, but that would just be extra work without yielding extra value.

I think this is something to be aware of when doing mutation testing. Not all escaped mutants are necessarily bad. For each of them you have to ask yourself the question “Would it be bad if I made this ‘error’ in my code?”. If the answer is no, don’t bother about the escaped mutant.

What test to add?

One of the main difficulties when trying to kill a mutant was figuring out what kind of test to add. Just from seeing a changed line in the code it is not always clear how to write a test that would fail on the given line. This was further amplified by the fact that the codebase I worked with was written by a colleague, and I did not know it inside out yet.

I learned that the HTML code coverage report generated by PHPUnit can be of tremendous help here. If you hover over a covered line of code there, it shows you which tests cover that line. This way you can lookup which tests already exercise the mutated line of code. The test you want to add to kill the mutant is probably a variation of one of them. This reduces your problem to analyzing these ‘example’ tests and reasoning about what you could change in them to fail when the mutation is present.

It improves your code too

Not only the test suite got some updates during my experiments with mutation testing; the production code improved as well. Sometimes the mutants generated by Infection were actually better than the original version! One such case looked somewhat like this:

<?php
class SomeStore {
    public static function createWithInMemoryDatabase(): self {
        $database = $database ?? new SqliteDatabase(':memory:');
        // ...
    }
}

Among the generated mutations there was one that removed the $database ?? null coalesce operator. This is actually an improvement, as the null coalesce operator is useless here! At the start of the function, $database is always null, so the operator always resolves to its right-hand side, creating a new database. This code was an artifact from a moment when the method was named differently and allowed injecting a custom database through a $database parameter. Now that parameter has been removed, we can get rid of the null coalesce as well. While other static analysis tools could have found this dead code as well, at least Infection brought it to our attention.

Another example where the mutant turned out to be better was the removal of a trim() function. At that spot in the code there could never be any significant or problematic whitespace. The trim()-call thus was unnecessary and could be removed.

Conclusion

Once you’ve had some practice with mutation testing, it can really help with improving both your test suite and code. Infection is straightforward to setup and use, and makes it fairly simple to get started in PHP. Just keep in mind that not all escaped mutants are a problem and blindly striving for 100% MSI does not add value. Try it out, and let me know what your experiences are!