Visual unit tests

I hope that most of us use unit tests in day to day development because it saves a lot of time for us to make something new instead of repeat the same mistakes again. In this article, I'll talk about our approach to deal with visual tests

Retrospective and wishful thinking.

Standard approach

All visual tests* based on the same approach.

• You should run the server with your application.
• You should write tests which run by NodeJS
• As a glue between our application and tests, we use Puppeteer or Playwright

Pros:

• Expectation code is clean and straightforward

const image = await page.screenshot();
expect(image).toMatchImageSnapshot();

Cons:

• Test code and code under tests are too far from each other In the test code, we usually call the driver to open some page where located code under test. In the best cases, such pages are somehow automatically generated. This led to notable switch between code under test and test itself that lead to frustration

  // Average visual regression test code
  const page = await browser.newPage(); // create a tab/page
  await page.goto('https://localhost:3000'); // navigate to URL where located the code which we want to test

• Complicated setup and teardown. To run tests, we usually should set up two servers. One for code under test and another for tests itself
• Usually such tests are slow due to complex setup and communication between test and code

All these problems and consequences prevent to write visual tests with pleasure. But visual problems haven't gone anywhere

What we want or ideal visual test for us

I always dreamed to have something similar to unit test experience. Where you can easily add or remove test. Play with code under tests by tweaking different options and observe the result. Focus on specific test and enjoy fast feedback loop between change and result

  render(<UserProfile/>) // code under test
  // test code
  const image = await page.screenshot(); // Take a screenshot
  expect(image).toMatchImageSnapshot(); // Test with reference

In the example above, our test code and what we want to test placed near each other. So by removing test, we're also removing code under test. We can experiment with test and code under test. Because it looks and behave as a regular unit test

Solution

For our unit tests, we use Karma because it's flexible, fast and solid tool to test frontend JavaScript in browser even after ~10 years of existing. Also, Karma would be our foundation for visual unit tests. So all approaches and tools which we can use and apply for unit tests are work for visual unit tests too. What we need is provide function to make a screenshot and matcher which compare reference with result.
To make it possible, we take Puppeteer or Playwright as launcher for our tests where we expose screenshot functionality and possibility to compare screenshot with reference image on the disk for our test code through expose function API.

We liked matcher functionality provided by jest-image-snapshot, so we just took this solution and adopted for jasmine and Karma.

The result is the boom of visual unit tests in our product. Because now write visual unit test as easy as write plain unit test

Conclusion

We love this approach because it already brings benefits of visual tests. And it does not require much effort. We pack it as a NPM package for karma, but what we love is the idea of having test code and code under test together, so you can look at our approach and maybe bring a more powerful tool. The repository itself contains tests, so you can open it via Gitpod or GitHub Codespace and play with it without additional setup.

• - When we're building these tool there no cypress which such functionality but now as I know it provide something similar, and this is cool. Maybe in the future we would migrate to it, but for now we are happy with our current approach

Pitfalls

Font

An innate problem of visual test is a drawing font on different platforms. For us, we decided that in visual tests we want to test only visual part and text/font is more logical part, so we just create a custom font for visual tests where all symbols just black squares which works on all required platforms for us. So the developer can easily write a visual test on macOS and make a reference, and it would work for Linux.

Different browsers

Currently, we use only Puppeteer/Chrome to simplify the process of installation, reduce headache with different browsers and improve developer experience to encourage writing visual unit tests. It works for us because it already brings benefits. But solution support use of Playwright(Chromium/Edge, Safari, Firefox) so in the future if all would work fine and robust we are planing to run our tests on different browsers