How to make PHPUnit Code Coverage 2+ times faster with Pcov compared to Xdebug

PHPUnit is the de-facto testing library for PHP. With the use of PCov you can speed up PHPUnit code coverage by 2-5 times for PHP 7.0+ application. In this post, we will compare the results of an experiment I did on Laravel framework tests. The tests were run without coverage, then with Xdebug coverage, and finally with PCov all on Github actions. Pcov took half the time to run the PHPUnit tests with code coverage compared to Xdebug, let’s go to the numbers.

Save time on CI builds

What does saving say 12 seconds on each build mean on the long run? If you can save just 12 seconds on each build, it equates to 1 minute in just 5 builds.

In just 100 builds you save 20 minutes and in 1000 builds that number becomes 3 hours and 20 mins.

Think of it as how much it will save in terms of waiting time for your colleagues too.

PHP code coverage with Pcov, not Xdebug

XDebug is a debugger that can do coverage too. PHPdbg is another alternative to Xdebug. Pcov is built for PHPUnit code coverage, not something else.

Michael Dyrynda has also talked about this issue, where he mentions:

You have access to the same output formats that are available to PHPUnit (formatted output, clover, JSON, HTML, etc.) with none of the overhead.

He has also pointed out other issues with Xdebug like hitting the max_nesting_limit too, you should read his blog post too.

Considerations for code coverage with PCov

• Pcov is a self-contained PHP code coverage driver for PHP 7 and above
• PHPUnit 8 and above supports PCOV out of the box, for PHPUnit 7 and lower you will need pcov-clobber
• Pcov makes perfect sense in a Continuous Integration (CI) environment as you don't debug code on a CI :)

To show a real-life scenario we are going to see how long Laravel Framework’s 5700+ tests with 15500+ assertions are going to take in our quick experiment.

Let’s get cracking!

Procedure for faster PHPUnit code coverage

I picked up the Laravel framework not only because it is very popular but also because there were a lot of tests, more than 5700 of them. On top of it, for the Laravel 8.x branch, the tests are running on Github Actions.

Tests for Laravel 8.x run on multiple versions of PHP like 7.3, 7.4,8 on lowest to stable variants. The same tests also run on windows.

Another reason to choose Laravel 8.x was it is using PHPUnit 9.3 which does not need pcov-clobber to get the PHPUnit coverage.

I had blogged about getting started with Unit testing in Laravel in the past which should be a good unit testing refresher. Data provider for PHPunit is also a great way to write less test code but achieve more code coverage.

Below are the steps I took to find out how fast Pcov was against Xdebug for PHPUnit code coverage.

Fork Laravel/framework repo and run tests only for PHP 7.4

To keep things simple, I forked the Laravel/framework Github repo. After that, I change the Github Actions tests workflow to run tests only on PHP 7.4 which the current stable version. You can see the changes I made in this pull request.

Opening the pull requests ran the tests without PHPUnit code coverage and it took 33 seconds to run the tests consuming 257MB of memory.

You can view the details of that test run in this Gitub Actions page, below is a quick screenshot.

Run PHPUnit code coverage with XDebug

I merged the above pull request to run tests only for PHP 7.4. Then I made changes to run the PHPUnit tests with code coverage using Xdebug as the driver. The change is very easy as Gitub action was using shivammathur/setup-php@v2 action. After a bit of Googling, I found that that action had Code Coverage support and it was very easy to enable.

I had to change the coverage from none to xdebug and add --coverage-text to the PHPUnit command making it:

vendor/bin/phpunit --verbose --coverage-text

I made those changes in 2 places in the tests.yml file and that resulted in this pull request. A new pull request = the tests running again in Github Actions CI.

With Xdebug code coverage I did a couple more runs to see if the time taken to run the test vary by much. It was generally the same.

In one of the runs of code coverage with Xdebug took 2 mins 34 seconds and consumed 395 MB of memory.

I am only checking the time for the Execute tests task. You can view the screeshot below:

The code coverage was reported as below, with 75.65% of the lines covered and 68.90% of the methods covered by PHPUnit Code coverage using Xdebug.

PHPUnit Code Coverage with Pcov is 2x faster

Now with the time of 154 seconds for Xdebug, I wanted to see how long it would take the new coverage driver Pcov. To find this out, I again followed a similar approach, went to the Laravel 8.x branch, and started editing the .github/workflows/tests.yml file. I change the coverage from none to pcov thankfully the PHP action supports pcov.

The changes I made are in this pull request. This triggered another build on Github Actions.

This time surprisingly it took just 1 minute 17 seconds and consumed 393 MB of memory.

Again this is for the Execute tests task as seen below:

Same as Xdebug the PHPUnit Code Coverage was reported as 75.65% of the lines and 68.90% of the methods covered by PCov. You can see other test runs in the Actions tab of my Laravel Framework fork.

Quick comparison of Code coverage

Let’s take a quick look at how long the PHPUnit test took with and without code coverage:

It is very clear that Pcov took half the time as Xdebug and even consumed lesser memory. PHPUnit code coverage with Pcov took 77 seconds and with Xdebug took double of that at 154 seconds.

In my local run in a docker container, the results were pretty different. For Xdebug these Laravel framework tests took 15 minutes 15 seconds (403 MB memory) and the with Pcov driver the same tests took 3 minutes 25 seconds (399 MB memory).

Pcov was 4.43 times faster on my local machine inside a docker container.

Not only me, but Swashata Ghosh has also reported a 5 times faster code coverage with Pcov in place of Xdebug. Speaking of pure numbers, I ran a check on a test where it took 17 seconds with Xdebug and it took just 1 second with PCov. It was 17 fold faster but that should not be a yardstick to compare Xdebug and Pcov for code coverage.

I did not try PHPDbg as an option because it was not available in the PHP Github action. If you want to quickly switch between Xdebug and Pcov please read this guide.

Conclusion

PHPUnit code coverage is usually coupled with Xdebug. It has a problem the code converge reports are slow with XDebug. PCov is purpose-built for PHPUnit code coverage not debugging and it makes gathering code coverage a lot faster.

If you want to speed up your code coverage and save time on your CI builds use Pcov in place of Xdebug. You will surely like the time saved after the process is done. Happy faster testing and coverage reports!