Visual Regression Testing with Cypress 10

In this week’s post join me as I battle with some visual regression plugins for Cypress component testing to find the one which actually works. Spoiler alert - it’s not perfect either.

Blown away by the Beta release of Cypress component testing (as part of Cypress 10) I wondered if it would be possible to also have a component visual regression test.

I started by checking out what Cypress docs have to offer. The docs tell that Cypress does not include this ability as a feature, but since you can write custom plugins for Cypress, one can write a plugin which does just that - compares image snapshots.

One plugin which Cypress suggests is the cypress-plugin-snapshots, but if you wander into Cypress’ plugin “market” you can see that there are more.
After checking a few, I have decided to go with the cypress-image-diff-js plugin because it’s free, has a relatively simple api, seems to have a good update activity and a fair amount of GitHub stars.

I will attempt to introduce this visual regression testing to my Pagination component from the @pedalboard/components package. So here we go.

I already have Cypress component testing installed and enabled on my repo as described here, so let’s start with adding the plugin to my dependencies:

yarn add cypress-image-diff-js -D

Following their “Cypress integration” docs I first need to add the plugin to Cypress, but the example given is not compatible with Cypress 10. Let’s change that so it comply with it - on our cypress.config.ts we add it like so (see the “setupNodeEvents”):

import {defineConfig} from 'cypress';
import getCompareSnapshotsPlugin from 'cypress-image-diff-js/dist/plugin';

export default defineConfig({
   component: {
       devServer: {
           framework: 'react',
           bundler: 'webpack',
       },
       setupNodeEvents(on, config) {
           getCompareSnapshotsPlugin(on, config);
       },
   },
});

Now we need to register the new command to Cypress so it can be called from the tests. Since the command configuration was also changed in version 10, we need to add these lines in the cypress/support/commands.ts file:

import compareSnapshotCommand from 'cypress-image-diff-js/dist/command';
compareSnapshotCommand();

Just to make sure we didn’t break anything, let’s try and launch Cypress… yeah, it appears to be running as before. Now it is time to see if we can add the visual regression test to our existing test.

Here is how my test looks like now - the line with compareSnapshot is the one I’ve added in order to take the snapshot of the component. The string is the name of the snapshot and the number next to it is the tolerance threshold, which is currently zero:

describe('Pagination component', () => {
   describe('PREV button', () => {
       it('should be disabled when reaching the first page', () => {
           mount(<Simple />);

           const prevButton = cy.get('button').contains('PREV');

           prevButton.click();
           prevButton.click();
           prevButton.click();

           prevButton.should('be.disabled');

           cy.compareSnapshot('Pagination', 0.0);
       });
   });
});

I run the test and sure enough a new snapshot appears under the baseline directory. We also see that we have a comparison and diff directories currently empty, and a visual-report directory dir as well:

Here is my component snapshot:

These sorts of images, the baselines, are the ones that you need to commit into your source control, since the comparison is always done between them and the new generated ones.

Ok, now for the real thing - I’m going to present a brutal visual change to the component and see if the snapshots detected that as a visual regression.
I define that the disabled button will be in “aqua” color. I will make the change in the component’s CSS file and check the snapshots afterwards -

The test runs again but now it fails with this message:

Error
Image difference greater than threshold: 0

You can obviously change the tolerance threshold (even for each test separately, which gives a lot of flexibility) but there is no doubt here that the component suffers a major regression.

How do our snapshots look now?
Now I can see the additional images in the comparison and diff directories. The interesting one is the one in the diff directory:

Yes, we can clearly see the issue here. Anything in the reports dir?
Hmm…. nothing. Weird 🤔

Let’s say I approve this change, how do I update the snapshot?
I can update ALL the snapshots with the CLI command `cypress-image-diff -u’, but this is not the best option to go with. I think that a better way is to just delete the baseline snapshots you would like to update, and the plugin will take care of the rest.

Wrapping up

It’s nice, it’s free but… it’s still not quite there.
While other plugins fail to work completely in Cypress 10, this one works but opens the browser in a high resolution (at least on my machine) and not having an easy way to update selected snapshots is disappointing.
Having said that, it might be a good solution for those who do not want to pay extra money for 3rd party vendors which charge by the snapshot (cough, Chromatic, cough).

As always if you have any suggestion on how to make this better, comments or questions - be sure to leave them in the comments below :)

Hey! If you liked what you've just read check out @mattibarzeev on Twitter 🍻

Photo by Jason Dent on Unsplash