DEV Community

Cover image for Storybook and Chromatic for Visual Regression Testing
jen chan
jen chan

Posted on

Storybook and Chromatic for Visual Regression Testing

This post continues some thoughts around unit testing components in design systems.

To boost your component library's test coverage through development, each component should have a visual snapshot in addition to unit tests. The unit tests serve as a form of documentation for the expected behavioural outcomes of your component; snapshots help validate the before and after of expected visual changes.

Compared to the unit test setup in my last post, Chromatic is a breeze. It only takes 2 min to add to a project and integrates easily with Github and Bitbucket repos.

Within "snapshotting", we'll discuss:

  1. DOM snapshots that capture changes in component markup
  2. Visual browser-based snapshots which I delegate to Chromatic in this post.

DOM Snapshots

DOM Snapshotting highlights exact changes to code, but it won't visually capture any regressions.

@storybook/addon-storyshots can be plugged into the .storybook/main.js. Along with the Jest setup I mentioned in my last post, we can do this to set up snapshotting:

// storyshots.test.ts

import initStoryshots from '@storybook/addon-storyshots';
initStoryshots();
Enter fullscreen mode Exit fullscreen mode

"DOM snapshots are awkward because it's tricky to determine > how a UI renders by evaluating an HTML blob."
Storybook Visual Testing Handbook, 2021

In fact, if you use a style framework like styled-components, unique class names are generated on component markup for compilation so you'll get meaningless inline changes that need to be updated.

Such changes may be flagged as errors on a CI pipeline and break it. Regressions or not, we want our pipeline to store the results, and for Chromatic to help us deal with the tests!

Automate DOM Snapshotting on a CI Pipeline

So I appended the following flags to my Jest script on my Bitbucket Pipeline:

jest --ci --passWithNoTests -u

Here, my package.json script is told to run npm run test:ci as jest --ci

The --ci flag tells Jest that it's running in a pipeline. The alias --passWithNoTests tells the test suite to complete if no test files are found. The alias -u flag stands for --updateSnapshot, thus telling our setup to take a new snapshot to replace the existing snapshot from a previous build. (Yes, you can push the snapshots to dev as a baseline to be compared future changes against.)

The above takes the snapshots, but to start activating visual snapshots with Chromatic you run :
npm run chromatic --project-token=YOUR_PROJECT_TOKEN

Screenshot of Bitbucket pipeline running Chromatic visual snapshotting tool with successful result

The flag, --exit-zero-on-changes will tell Chromatic to stop running if there are no actual changes to your components, (or it'll keep looking and the process will not finish, your pipeline will timeout or not complete).

Visual Snapshots for Regression Testing

Chromatic's free tier offers 5000 free Chrome snapshots per month. This should be enough to get you covered for the first months on a new project, but if you inherited a legacy design system, that's questionable since it's prone to take a snapshot of every single component unless you use their option to exclude snapshots on certain stories. And if you're inheriting a legacy design system then you, my fellow developer, should advocate to tool/automate for developer and QA efficiency. It's the right decision for a product.

The codeblock I included previously is just the start to making Storybook do DOM snapshots. If you want to take a programmatic route to visual testing, Storybook's add-on @storybook/addon-storyshots-puppeteer offers device emulation. I haven't used this on production, but from what I've seen, you're able to write full suites of end-to-end device tests!

But why waste time on that when you can tell Chromatic to capture different viewports? (Ok yeah I get it, there's no match for actual device testing, since devices actually have different pixel density ratios and auto zoom.)

Whether you automate this or decide to run it locally, on completion, Chromatic will supply a link to the latest version of a published Storybook and provide a link to click through to the UI Review dashboard.

Chromatic Dashboard

One of the gotchas is that you need to commit and run on the same branch to capture the "before" and "after" of visual component changes. That means you need to have a history of running Chromatic on your main or a specific PR for every commit you make.

Screenshot of Chromatic Visual Regression testing software highlighting visual differences between previous code commit and next code commit

Teammates can be added as developer, viewer, and reviewer roles who can "Accept" or "Deny" the changes presented in snapshots.

It's best used with designers and QA folks have have the capacity to perform UI reviews. Here's me, reviewing my own.

Screenshot of webhooks notifying tech team of updates to visual snapshot tests
That's a Slack hook for build and UI Review results.

It seems extra but when there's many components that need small changes, this quickly becomes helpful for making developer acceptance testing faster.

Summary

Chromatic is a (primarily visual) snapshotting tool that takes the pain out of setting up visual snapshotting and regression tests once you've overcome the hurdle of setting up Jest with Storybook.

Until we train AI to calculate the congruence of built UIs matching a design pixel-perfectly or the animated prototype perfectly... we'll still need humans to identify visual regressions.

Assuming you have buy-in and dedicated resources for development of design systems, then Chromatic is a very useful tool that would save you the repetitive activity of going through BrowserStack for cross-browser testing.

Relevant Reading

Discussion (0)