Visual regression tests for Storybook with Loki

How to setup visual regression tests with Loki for your design system

To my knowledge, there are currently no leading standalone tools for visual regression testing. Many companies implement their own solutions, but there are no tools that are widely used and recommended as a standard.

Some companies actively explore OSS solutions and in this article, I will share my experience with one of them.

What is Loki?

Loki is a tool that makes it easy to test Storybook projects for visual regressions. Loki was used for the trial stage at Thoughtworks tech radar.

Why Loki?

There are a few visual regression tools for the web, but most either cannot be run headless or use phantomjs which is deprecated and a browser nobody is actually using. They usually also require you to maintain fixtures. With react-native it's now possible to target multiple platforms with a single codebase, but there's no single tool to test all to my knowledge.

How to setup Loki for your project?

1. Install Loki package

npm i loki --save-dev

or if you use yarn

yarn add loki -D

2. Create a Loki config file

yarn loki init

This will add standard configuration to your project. You can find it in the package.json file.

Example of the configuration:

{
  "name": "my-project",
  "version": "1.0.0",
  "loki": {
    "chromeSelector": "#my-decorator > *",
    "configurations": {
      "chrome.laptop": {
        "target": "chrome.app",
        "width": 1366,
        "height": 768
      },
      "chrome.iphone7": {
        "target": "chrome.docker",
        "preset": "iPhone 7"
      },
      "chrome.a4": {
        "target": "chrome.aws-lambda",
        "preset": "A4 Paper"
      },
      "ios": {
        "target": "ios.simulator"
      },
      "android": {
        "target": "android.emulator"
      }
    }
  }
}

I prefer not to pollute my package.json and extract configuration to a separate file loki.config.js:

// loki.config.js
module.exports = {
  diffingEngine: 'pixelmatch',
  configurations: {
    'chrome.laptop': {
      target: 'chrome.docker',
      width: 1366,
      height: 768,
      deviceScaleFactor: 1,
      mobile: false,
    },
    // 'chrome.iphone7': {
    //   target: 'chrome.docker',
    //   preset: 'iPhone 7'
    // }
  },
}

I'll go with the simplest configuration for now, but you can find more information about the configuration options in the Loki documentation.

I choose the pixelmatch engine for now, but you can also choose between GraphicsMagick (gm) and looks-same. Currently, three diffing engines are available. Worth to mention that if you'd like to use graphicsmagick diffing engine,
for image comparison, you need to install it on the machine / CI as an additional dependency.

3. Install Docker

Loki uses Docker to run tests in an image. If you don't have Docker installed, you can follow the installation instructions.

4. Start Storybook

When running tests locally, you need to run a local instance of the Storybook & start Docker to execute the tests.

yarn storybook

5. Generate reference images

yarn loki update

will create a reference image for each story in your Storybook & place them in the .loki/reference directory.

6. Run tests

yarn loki test

or if you want to run tests for a specific platform (e.g. laptop, iphone7 etc.)

yarn loki test laptop

On each test run, Loki will create new files in the .loki directory. Inside this directory, you will find the current, difference, and reference folders.

• The current directory contains the images that were generated during the test run.

• The difference directory contains the images that were generated during the test run and are the ones that were different from the reference images with, visual differences shown.

• The reference directory contains the reference images that were generated during the loki update command. These are the images that are used to compare against the current images.

Images from the reference directory should be committed to the repository if you want to use them as a reference for your tests on CI.

7. Add Loki to CI (as GitHub action)

To run Loki on CI you need to add a new job to your CI pipeline.

Update NPM scripts

{
  "scripts": {
    "visual-tests": "loki test laptop",
    "visual-tests-ci": "build-storybook && loki --requireReference --reactUri file:./storybook-static"
  }
}

Add a new job to your CI pipeline as a GitHub action

# .github/workflows/ci.yml

visual-regression-tests:
  name: 'Visual tests: React DOM'
  runs-on: ubuntu-latest

  steps:
    - uses: actions/checkout@v3
    - name: Setup node from node version file
      uses: actions/setup-node@v3
      with:
        node-version: '16'
        cache: 'yarn'
        registry-url: 'https://npm.pkg.github.com/'
        scope: '@your-org'
    - name: Install dependencies
      run: yarn --frozen-lockfile --non-interactive --silent --ignore-scripts
      env:
        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
    - name: Run loki
      run: yarn visual-tests-ci
    - name: Archive screenshots
      if: ${{ failure() }}
      uses: actions/upload-artifact@v1
      with:
        name: design-system
        path: .loki

Skipping stories

If you want to skip some stories from the tests you can add loki.skip parameter to the story:

<!-- Badge.stories.mdx -->

import { Canvas, Story } from '@storybook/addon-docs'
import Badge from './Badge'

<Canvas>
  <Story
    name="warning"
    parameters={{
      loki: {
        skip: true
      }
    }}
  >
    <Badge variant="warning">
      On hold
    </Badge>
  </Story>
</Canvas>

Approve changes

If you want to approve changes you can run:

yarn loki approve

This will update images in the reference folder if changes are intended and valid.

Accessing artifact files from GitHub action

In case of a flaky test or a test that fails on CI but passes locally, you can download the artifact files and run the tests locally to see what's going on.

To access artifact files you can go to the "Actions" tab in your repository, find your workflow build, navigate to "Summary" and download the artifact from the latest run:

Pros

• Relatively easy to setup
• Multi devices/platforms support
• Running in Docker
• Different comparison engines to choose from

Cons

• Building a Storybook on each test run, Loki requires it to perform tests on a static build output. The build takes time & resources on CI. A possible solution: cache this somehow
• No handy way to visually compare/accept differences/changes between the test runs

Comparison stats - VRT tests

Tests were run on a local machine and on a CI machine (build time excluded).

The tests were run on a Storybook with 30 components and on average 3 stories per component.

The tests were run with 3 different comparison engines.

There are three currently available options to choose from when comparing images in Loki.

Conclusion

It's important to have a visual regression testing tool in your tool belt. It's a great way to catch visual bugs early and prevent them from reaching production.

There are many things to consider when implementing visual regression testing, it should not slow down your development process and release process.

Although there are some cons to using Loki, I'd give it a try and see if it works for you. It's a decent tool for catching visual bugs and it's relatively easy to set up.

If you have any questions or suggestions, feel free to reach out to me on Twitter @alxgri