Improving Docker file performance on MacOS

Last year, we started switching our development environment at SpareRoom from a Vagrant/VirtualBox solution to Docker. This was to make a more robust environment that supported all our web-app territories at the same time, as well as support the Apple Silicon Macs we'd be upgrading to.

Docker Desktop on MacOS

While the Docker environment had many advantages, the performance on our MacOS dev machines came as a shock, and we quickly found out this was due to the dreadful file system performance (mainly for Docker volumes). The effect was very obvious, e.g. an Apache thread would take an extra couple of seconds just to read the app libraries. We were looking into implementing a Mutagen solution for syncing local/docker filesystems, which requires some setup but has helped other MacOS users considerably, when Docker Desktop v4.6 was released (March 2022) with an interesting new feature.

VirtioFS

Docker Desktop 4.6 added beta support for VirtioFS for MacOS Monterey and above. Just enabling it in the (beta at the time) settings screen and relaunching the containers was all needed for instant performance gains!

To be honest, the first time I enabled it, Docker Desktop simply got stuck and had to be killed. The Docker engine would not start (shown as "stopped") until I disabled the option directly in the settings file:

~/Library/Group\ Containers/group.com.docker/settings.json

(set useVirtualizationFrameworkVirtioFS to false)
But the culprit was soon found - I had migrated to a new Mac and in my Docker file sharing settings a no-longer existing directory was listed. If you have a similar problem, check in the above settings file in case there is a removed or inaccessible directory under filesharingDirectories. Apparently VirtioFS just hangs forever without errors in that case - yeah, I know, not the best programming practice around that!

On Docker Desktop v4.15, a bug was fixed where some users had file permission issues, but with it the VirtioFS performance degraded significantly. The current v4.17 is a faster than v4.15/4.16, but my advice is try v4.14.1, which is the latest "fast" version. If you have no issues with file permissions (it was not a widespread problem anyway), keep that until the performance regression is fixed.

Colima

As I was tracking the above performance regression, I discovered an open-source container runtime for MacOS/Linux called Colima. It is pretty much a Docker Desktop drop-in replacement and also offers VirtioFS (on Ventura - i.e. MacOS 13.x or newer). You don't need to uninstall Docker Desktop, just stop it and install Colima e.g. through Homebrew:

brew install colima

You should already have docker installed (otherwise brew install docker would suffice, and you create a VM to run the containers with the following command:

colima start --cpu 2 --memory 2 --vm-type=vz --mount-type=virtiofs

You can specify the CPU/RAM you prefer, and if you need a Rosetta 2-enabled vm, the option is --vz-rosetta. On the first run it will setup the VM to run your containers. After it finishes, the docker context show command should return colima, which means docker runs under Colima and you can use docker as usual under it.

The command colima stop will return docker context show to default, so starting Docker Desktop after that allows you to go back to it if you wish.

Colima seems to be even faster than Docker Desktop 4.14, however, I did encounter some stability issues - the project does specify that the vz/VirtioFS support is recent, so hopefully it will quickly mature. It is definitely worth trying out and keeping an eye on, especially since it is so simple to set up as an alternative.

Performance Comparison

I ran a couple of relatively file-access heavy comparison tests on the latest Docker Desktop for both "standard" gRPC-FUSE and VirtioFS, as well as the earlier 4.14.1 version and the latest Colima.

The first test is a collection of SpareRoom API tests without any module preloading, that have to load parts of the webapp and also access the mysql and sphinx search containers. The second is the test suite of the Moose 2.2203 Perl package, which is my go-to when I want an "overweight OO framework" test.
Note that these are far from pure file I/O tests: a file I/O-only test would show much more pronounced differences, but I don't like synthetic tests. Hence, I went as real-world as possible, for example the Moose test suite runs like that whenever you install the package with cpanm Moose.

Here are the results, running on an M1 Pro (arm64 containers):

VirtioFS was a game-changer, it is 2-3x faster in these tests compared to gRPC-FUSE, and I've seen even bigger differences than that. But, given that the Moose test is twice as fast running natively on the same M1 Pro (39s), there could still be improvements. Especially for Docker Desktop which is currently lagging not only behind Colima, but also from its own earlier versions. Colima on the other hand, while measurably faster does seem a bit immature - I've had to restart my environment due to issues etc. But given the fact the VirtioFS support is recent and under active development, it's definitely one to watch.