DEV Community

Cover image for Vitest vs Jest benchmarks on a 5 year old real work SPA
The Jared Wilcurt
The Jared Wilcurt

Posted on • Updated on

Vitest vs Jest benchmarks on a 5 year old real work SPA


The following is about a 5½ year old production web app with 1,256 real world unit tests across 139 test suites. This production app has been the primary daily job of ~5 engineers since it's inception. Building it out, updating it, adding/demoing features, etc. It's a rare, real-world example of a largish webapp.

Motivation to switch to Vitest

Due to a bug that has existed in Jest since 2020, my team at work had to decide to either, fix the bug in Jest's repo.... or switch over to Vitest which doesn't have the bug. And since Jest's repo would require someone on our team to deal with a large Yarn mono repo, and TypeScript, it seemed like it might be worth it to avoid going that route.

Notably, we were not concerned with performance. So these benchmarks are more just personal curiosity than anything. But still worth sharing.

Also worth a note, these benchmarks were ran on the "worst case scenario" machine (the one with the slowest results to emphasize the biggest difference). I will also mention the results on the best case scenario (our CI).

Converting over the smaller repo first

Before taking on the multi-week task of converting the larger repo over to Vitest, it made sense to do a sort of "trial run" on the smaller repo.

Smaller Repo Stats:

  • Vue 2 + Vuex + Vue-Router + Vite + Element-UI
  • App Routes: 6
  • Test Suites: 14
  • Unit tests: 147
  • Snapshots: 64
  • From: Jest + JSDOM
  • To: Vitest + HappyDOM
  • Added 3 new dependencies, removed 7
  • Benchmarks on slowest machine:
    • npm run unit (cold)
      • Jest - 35.79s
      • Vitest - 21.31s
    • npm run unit (warm)
      • Jest - 27.77s
      • Vitest - 16.38s
    • npm run unit-coverage
      • Jest - 25.39s
      • Vitest - 18.75s

So as you can see, Vitest actually outperformed Jest in all scenarios on the smaller repo. However, I think this had a lot more to do with switching to HappyDOM than Vitest itself.

Bigger Repo Stats:

  • Vue 2 + Vuex + Vue-Router + Vite
  • App Routes: 73
  • Test Suites: 139
  • Unit tests: 1,256
  • Snapshots: 517
  • From: Jest + JSDOM
  • To: Vitest + JSDOM
  • Added 3 new dependencies, removed 3
  • Benchmarks on slowest machine:
Tool Coverage Real-Time Self-Time Transform Setup Collect Tests Env Prepare
Vitest 320.90s 9.71s 82.14s 442.08s 325.19s 232.20s 56.45s
Vitest 318.71s 9.73s 83.86s 447.80s 295.65s 240.87s 57.98s
Vitest 332.02s 327.54s 8.97s 82.09s 455.03s 328.10s 241.90s 56.88s
Vitest 415.32s 408.03s 9.80s 99.78s 568.28s 420.10s 294.90s 70.65s
Vitest ✔️ 297.17s 9.47s 115.97s 638.88s 538.10s 370.47s 96.27s
Vitest ✔️ 272.07s 9.37s 100.96s 576.24s 540.31s 328.53s 84.68s
Vitest ✔️ 293.33s 287.45s 11.15s 111.67s 624.03s 551.18s 348.24s 92.34s
Vitest ✔️ 272.23s 265.44s 9.17s 102.19s 580.97s 491.47s 320.93s 83.75s
Jest 213.68s 207.69s
Jest 201.76s 191.56s
Jest ✔️ 226.06s 221.30s
Jest ✔️ 273.54s 267.91s
Vitest --no-isolate 115.94s 12.73s 6.48s 53.88s 264.15s 7.65s 2.61s
Vitest --no-isolate ✔️ 242.30s 10.53s 13.89s 121.78s 532.12s 33.38s 109.14s

Benchmark Results:

Note that "Real-time" was me timing the run with a stop watch and "Self-Time" is what the test runner reported as the total duration. The two are actually always within a few seconds of each other.

Average without coverage:

  • Vitest = 322s (+37.8% time) on average 122 seconds slower
  • Jest = 200s

Average with coverage:

  • Vitest = 281s (+12.8% time) on average 36 seconds slower
  • Jest = 245s

You'll note that this time around Vitest was actually a little slower than Jest. And very weirldy, Vitest with coverage reporting enabled is actually faster than with it off. I believe this is mostly due to not switching to HappyDOM. There were enough test failures that needed fixing that just swapping out Jest was enough work for ~2 weeks. So I decided to skip HappyDOM.

But how can we make it faster?

In my last post, I documented many ways to make Vitest go faster:

Here are some of the things from that post that I tried, and the results:

  • --single-thread and --no-threads didn't work. Basically no test would even run. This may just be the first test failing and the rest choked, but I didn't want to bother to find out.
  • Both css: false and useAtomics: true actually made it slower
  • With --no-isolate it was 2.8x faster than vitest and 1.7x faster than Jest, but 19 tests failed (see table above). Some people report issues with watch mode when using --no-isolate. So I decided to not pursue it any further. Once the vm module that Vitest relies on supports ESM, or when the amazingly named Shadow Realms are added to JavaScript, we will likely get this performance boost for free without the downsides.

What about on CI?

All of our repo's are "branch protected" on GitHub. They must pass ESLint, SassLint, Build, Unit tests, and E2E tests.

When running on GitHub Actions (GHA), we see average times of:

  • Jest = 128s
  • Vitest = 138s

Because it's only a 10 second difference to run the full suite on GHA, and we primarily just run specific test suites when working locally. The difference in time is negligible. Also, we know that eventually the bottlenecks preventing Vitest from going faster will be removed and it will then start easily out-performing Jest.

So we are sticking with Vitest going forward. At least... once this 239 file PR with +10,486, -36,461 diff gets reviewed 😰.

Wait did you say Vue 2?

Yes, these projects are using Vue 2. Vue 3 did not exist in 2017/2018 when these projects were started. We have plans to convert the larger repo to Vue 3 by the end of the year. And we will be sticking with the superior Options API in all repos.

Though newer repos we've started use this boilerplate. I would recommend it to everyone, for every project.

Photo Credit:

Top comments (1)

nyambe profile image

There is a lot of legacy code out there that will need to go through this process, including some of mine. Thanks