4 Ways to Automate Visual Regression Tests

#webdev #javascript #react #testing

Before we get going, let's go over some quick questions:

What does visual regression testing mean?

Visual regression testing is a process of comparing the visual appearance of an application's UI between two different versions to ensure that there are no unintended visual changes. The process involves taking screenshots of various UI elements and comparing them to previously captured screenshots to identify any changes in the visual appearance. Visual regression testing helps to ensure that the application's UI remains consistent and visually appealing across different devices, browsers, and operating systems.

Why should you automate it?

Automating visual regression testing can help to identify UI issues early in the development cycle, which can save significant amounts of time and money that might otherwise be spent fixing issues later on. Think about the time (and therefore money) that is spent fixing UI quirks that are caught in Q/A, UAT, or production. Why wouldn't you automate catching those visual bugs?

What about manual visual checks?

Although you can perform manual visual checks, it's not recommended to solely rely on them because it takes significantly more time compared to using an automated suite. Spotting differences in the UI can be challenging, even for subtle visual bugs. See the example below - can you spot the 8 differences? Probably. But it'd take you a lot longer than an automation suite.

What percentage of visual bugs are caught by automated visual regression tests?

Studies from various sources (BrowserStack, AppliTools, etc.) show that automated visual regression testing can catch most visual bugs, with a success rate ranging from 95% to 99%, depending on the application's complexity and the testing process used.

In this article, we will explore 4 popular libraries for automating visual regression testing: Chromatic, Percy, Jest Image Snapshot, BackstopJS.

Chromatic

Chromatic is a visual regression testing tool that is designed specifically for Storybook. Chromatic provides a cloud platform which allows users to test and manage the UI components of a project, ensuring that any changes made do not result in unexpected visual changes.

Who uses Chromatic?

Adobe, Auth0, CircleCI, GOV.UK, BBC, etc.

Pros of Chromatic

Easy Integration: Having been built specifically for Storybook (by the maintainers of Storybook!), it's seamless to set up for your component library or design system.
Team Collaboration: Chromatic provides a platform for team collaboration by allowing you to share test results with your team members, leave comments, and discuss any issues.
Build Analysis: Chromatic provides detailed build analysis, showing you what changes were made, who made them, and how they affected your UI components.

Cons of Chromatic

Limited customization options: While Chromatic's UI is designed to be easy to use, it may not offer as much flexibility or customization as other visual regression testing tools.
Pricing: Chromatic offers a free plan with unlimited collaborators and storybooks. However, its free tier has a limit of 5,000 snapshots per month, so it's worth considering if the first tier ($149/mo) is affordable to your team if you think you'll hit that limit. Further pricing information can be found here.
No local testing option: Chromatic does not currently offer a way to test visual changes on a local machine before deploying them to the server, which could be a limitation for some teams.

Percy

Percy is an all-in-one visual review platform that offers a wide range of features and integrations. It was acquired by BrowserStack, which is an industry leader in application and browser testing. Percy helps developers catch visual regressions, streamline the review process and allows teams to easily collaborate and track changes.

Who uses Percy?

Google, Sentry, Shopify, Fastly, Basecamp, etc.

Pros of Percy

Versatile & Flexible: Percy can be used for visual regression testing in a wide range of applications beyond just Storybook. Percy has advanced features such as cross-browser testing, integrations with popular testing frameworks, and a powerful API that allows for more extensive customization and control over your testing processes.
Scalability: Percy scales very well, making it easy to manage visual regression tests for even the largest projects.
Collaboration: Percy provides a wide range of collaboration tools, including commenting and sharing features, that allow developers to work together to identify and resolve issues quickly.

Cons of Percy

Limited customization: Some users may find that Percy's out-of-the-box settings and features do not meet their specific testing needs, as there are limited options for customization.
Dependency on external service: Since Percy is a cloud-based tool, developers will need to rely on external service providers for hosting and maintaining the platform, which can introduce additional dependencies and potential points of failure.
Pricing: Percy offers a free plan with unlimited collaborators and storybooks. However, its free tier has a limit of 5,000 snapshots per month, so it's worth considering if the first tier is affordable to your team if you think you'll hit that limit. Further pricing information can be found here.

Which Cloud solution to choose for Storybook?

In my opinion, I would go with Percy if I were already using/planning to use it for other visual regression tests (e.g. in Cypress E2E test suites) so that your visual regression tests are consolidated in a single platform. On the other hand, if I only needed to perform visual regression tests for Storybook, I would opt for Chromatic. Chromatic is specifically designed for Storybook by its maintainers, making it more compatible and future-proof. Since Chromatic is optimized solely for Storybook, it integrates better with the platform and provides a more streamlined workflow.

The NPM trends of @percy/storybook vs. chromatic seem to indicate that most people have come to the same conclusion:

Jest Image Snapshot

Jest Image Snapshot is a visual regression testing tool built for Jest. It allows developers to take snapshots of the rendered components or elements and compare them with the previously stored images to check for any visual differences or changes. It works by rendering the component or element and generating an image of the rendered output using a headless browser. The generated image is then compared with the previously stored image using pixel-to-pixel comparison, and if any differences are found, Jest will report a failed test.

Who uses Jest Image Snapshot?

American Express, etc.

Pros of Jest Image Snapshot

Easy to Use: Jest Image Snapshot is easy to set up and use, making it accessible for developers of all levels of experience. It integrates seamlessly with Jest, a popular testing framework, and requires no additional configuration.
Independent: Jest Image Snapshot can be used locally, without the need for an external service, making it a more convenient and cost-effective solution for developers who prefer to test and debug their applications locally.
Pricing: It's an open-source project and it's 100% free.

Cons of Jest Image Snapshot

Performance overhead: Local visual regression testing can be resource-intensive, and running tests can slow down the development process, particularly for large-scale projects.
Dependence on Jest: As the name suggests, Jest Image Snapshot is tightly integrated with the Jest testing framework, which may not be suitable for teams using other testing frameworks.
Limited configuration options: While Jest Image Snapshot is easy to set up and use, it does not offer as many configuration options as other visual regression testing tools. This can be limiting for teams that require more flexibility in their testing.

BackstopJS

BackstopJS is an open-source visual regression testing tool for web applications that automates the process of comparing screenshots of the current state of a web application with its previous version to identify any visual changes. It uses headless browser automation to simulate user interactions and captures screenshots of web pages. BackstopJS can be configured to test multiple pages and responsive design across multiple viewports. It also supports user authentication and can be integrated with various CI/CD pipelines. BackstopJS uses a configuration file to define the URLs to be tested, viewports, and other settings.

Pros of BackstopJS

Flexible configuration: BackstopJS is highly configurable, which makes it suitable for testing different types of applications with various visual requirements. It allows you to define custom scenarios, set up different environments, and adjust parameters to achieve the desired level of testing.
Powerful comparisons: BackstopJS provides a powerful image comparison engine that detects even the smallest visual differences. It uses pixel-by-pixel comparison and threshold values to determine whether a change is significant enough to be considered a failure.
Pricing: It's an open-source project and it's 100% free.

Cons of BackstopJS

Complexity: BackstopJS requires more configuration and setup compared to other visual regression tools, which may make it more complex to implement.
Slower testing: BackstopJS can be slower to run compared to other visual regression tools due to how it generates and compares screenshots.
Limited support: While BackstopJS has documentation and community support available, it may not be as robust as other tools with larger user bases and more established support networks (e.g. Percy).

In conclusion, each tool offers unique features and benefits, as well as potential drawbacks. The choice of which to use for visual regression testing depends on various factors such as project requirements, team preferences, and available resources. There are trade-offs in selecting each tool, and the decision depends on what factors are most important for your project.

In my opinion Percy offers the most comprehensive and user-friendly solution, making it an excellent option for teams looking for an advanced tool. Chromatic is a suitable option for teams only needing to test their Storybook library, while BackstopJS is better suited for teams looking for more comprehensive testing capabilities. Jest Image Snapshot is a good option for developers working on smaller projects which use Jest.

Ultimately, incorporating automated visual regression testing into your development process is critical to catching UI defects early and ensuring a high-quality user experience for your website or application.