Vitest vs Jest benchmarks on a 5 year old real work SPA

Intro

The following is about a 5½ year old production web app with 1,256 real world unit tests across 139 test suites. This production app has been the primary daily job of ~5 engineers since it's inception. Building it out, updating it, adding/demoing features, etc. It's a rare, real-world example of a largish webapp.

Motivation to switch to Vitest

Due to a bug that has existed in Jest since 2020, my team at work had to decide to either, fix the bug in Jest's repo.... or switch over to Vitest which doesn't have the bug. And since Jest's repo would require someone on our team to deal with a large Yarn mono repo, and TypeScript, it seemed like it might be worth it to avoid going that route.

Notably, we were not concerned with performance. So these benchmarks are more just personal curiosity than anything. But still worth sharing.

Also worth a note, these benchmarks were ran on the "worst case scenario" machine (the one with the slowest results to emphasize the biggest difference). I will also mention the results on the best case scenario (our CI).

Converting over the smaller repo first

Before taking on the multi-week task of converting the larger repo over to Vitest, it made sense to do a sort of "trial run" on the smaller repo.

Smaller Repo Stats:

• Vue 2 + Vuex + Vue-Router + Vite + Element-UI
• App Routes: 6
• Test Suites: 14
• Unit tests: 147
• Snapshots: 64
• From: Jest + JSDOM
• To: Vitest + HappyDOM
• Added 3 new dependencies, removed 7
• Benchmarks on slowest machine:
  • npm run unit (cold)
    • Jest - 35.79s
    • Vitest - 21.31s
  • npm run unit (warm)
    • Jest - 27.77s
    • Vitest - 16.38s
  • npm run unit-coverage
    • Jest - 25.39s
    • Vitest - 18.75s

So as you can see, Vitest actually outperformed Jest in all scenarios on the smaller repo. However, I think this had a lot more to do with switching to HappyDOM than Vitest itself.

Bigger Repo Stats:

• Vue 2 + Vuex + Vue-Router + Vite
• App Routes: 73
• Test Suites: 139
• Unit tests: 1,256
• Snapshots: 517
• From: Jest + JSDOM
• To: Vitest + JSDOM
• Added 3 new dependencies, removed 3
• Benchmarks on slowest machine:

Benchmark Results:

Note that "Real-time" was me timing the run with a stop watch and "Self-Time" is what the test runner reported as the total duration. The two are actually always within a few seconds of each other.

Average without coverage:

• Vitest = 322s (+37.8% time) on average 122 seconds slower
• Jest = 200s

Average with coverage:

• Vitest = 281s (+12.8% time) on average 36 seconds slower
• Jest = 245s

You'll note that this time around Vitest was actually a little slower than Jest. And very weirldy, Vitest with coverage reporting enabled is actually faster than with it off. I believe this is mostly due to not switching to HappyDOM. There were enough test failures that needed fixing that just swapping out Jest was enough work for ~2 weeks. So I decided to skip HappyDOM.

But how can we make it faster?

In my last Dev.to post, I documented many ways to make Vitest go faster:

• https://dev.to/thejaredwilcurt/improving-vitest-performance-42c6

Here are some of the things from that post that I tried, and the results:

• --single-thread and --no-threads didn't work. Basically no test would even run. This may just be the first test failing and the rest choked, but I didn't want to bother to find out.
• Both css: false and useAtomics: true actually made it slower
• With --no-isolate it was 2.8x faster than vitest and 1.7x faster than Jest, but 19 tests failed (see table above). Some people report issues with watch mode when using --no-isolate. So I decided to not pursue it any further. Once the vm module that Vitest relies on supports ESM, or when the amazingly named Shadow Realms are added to JavaScript, we will likely get this performance boost for free without the downsides.

What about on CI?

All of our repo's are "branch protected" on GitHub. They must pass ESLint, SassLint, Build, Unit tests, and E2E tests.

When running on GitHub Actions (GHA), we see average times of:

• Jest = 128s
• Vitest = 138s

Because it's only a 10 second difference to run the full suite on GHA, and we primarily just run specific test suites when working locally. The difference in time is negligible. Also, we know that eventually the bottlenecks preventing Vitest from going faster will be removed and it will then start easily out-performing Jest.

So we are sticking with Vitest going forward. At least... once this 239 file PR with +10,486, -36,461 diff gets reviewed 😰.

Wait did you say Vue 2?

Yes, these projects are using Vue 2. Vue 3 did not exist in 2017/2018 when these projects were started. We have plans to convert the larger repo to Vue 3 by the end of the year. And we will be sticking with the superior Options API in all repos.

Though newer repos we've started use this boilerplate. I would recommend it to everyone, for every project.

---

Photo Credit:

• "brown wooden bench near body of water" - Anthony Cantin