DEV Community

loading...
Cover image for Why our visual regression is so slow?

Why our visual regression is so slow?

dmtrkovalenko profile image Dmitriy Kovalenko ・4 min read

Hola! Lazy dev here, and today we are going to discuss computer eyes πŸ‘πŸ‘ that help us to test how our products look like.

The visual regression tools that we are using every day are extremely slow. That is the fact. At least because the task to compare 2 images is hard. And I tried to fix this πŸ‘€.

But first of all, let's take a look at how visual regression tools work under the hood and what they are actually doing.

Image comparison

Image comparison itself is pretty hard. We can not just compare 2 image buffers using "===" and even we can not take all the pixels from the image and compare them one by one. Why? Because human eyes can not see the difference between all the colors. For example, how do you think – do these colors are the same or not?

Alt Text

If yes – πŸ‘ you have a really good vision! But 95% of people will not spot the difference between rgba(200, 100, 100, 250) and rgba(200, 100, 95, 250) colors.

That's why in order to visually compare images – we need to take all the pixels in the image and it is a lot (for a full-HD screenshot (1920x1080) we have 1920 * 1080 = 2 090 880 pixels) – iterate them one by one and calculate the color difference.

This is a hard task for computers. So the algorithm is:

1) Read and decode the image
2) Make a loop with millions iterations, make some calculations and save the different pixels
3) Make an image of different pixels
4) Encode and save the image diff

But the result is extremely helpful when you are testing the user interface:

Alt Text

The slowness

But unfortunately, tools that we are commonly using to compare screenshots are extremely slow. And they are slow not because they had a bad code inside. The main problem is that they are often written in the wrong language* or doing some useless job under the hood.

For example, the most popular image comparison tool in the javascript community – pixelmatch is really slow in Node.js environment (but blazing fast in the browser otherwise).

Using pixelmatch to comparing 2 screenshots of https://cypress.io home page will take around 7-8 seconds.

Alt Text

Impact on your CI

This really affects our CI time. Let's calculate the CI time for visual regression if we are running 25000 screenshot tests per month. And this number is not something overwhelming. It is a very basic plan of https://percy.io, which is usually not enough for huge projects.

So, if we are running 25000 visual tests and each screenshot test running for 7 seconds we are spending 48,6 hours on CI!

25000 * 7 / 3600 = 48,611111111
Enter fullscreen mode Exit fullscreen mode

That is a lot! This can take even more time than all the other UI tests and that's why performance for this sort of task really matters.

Cause if we can save at least 3 seconds per each snapshot we will save 20 hours per month.

25000 * 3 / 3600 = 20,83333 
Enter fullscreen mode Exit fullscreen mode

Solution

That was a pain point for me, so I decided to fix this – and wrote that fastest in the world image comparison tool.

And I am happy to introduce you odiff! πŸ‘€πŸ₯³πŸŽ‰ It was designed to handle the "huge" images, be fast, memory-efficient, and save your CI time.

Alt Text

This tool is running the same comparison 2 times faster than analogs! Yes, it can save you those 3 seconds per snapshot :)

Here are some benchmarks:

Alt Text

Also, here are the results of comparing the same cypress.io home page screenshot:

Command Mean [s] Min [s] Max [s] Relative
pixelmatch www.cypress.io-1.png www.cypress.io.png www.cypress-diff.png 7.712 Β± 0.069 7.664 7.896 1.82 Β± 0.03
ImageMagick compare www.cypress.io-1.png www.cypress.io.png -compose src diff-magick.png 8.881 Β± 0.121 8.692 9.066 2.09 Β± 0.04
odiff www.cypress.io-1.png www.cypress.io.png www.cypress-diff.png 4.247 Β± 0.053 4.178 4.344 1.00

How?

Why it is so fast? The answer is simple:

  • It is written in OCaml and compiled to the native binary executable. OCaml compiler is extremely fast and predictable so it is easy to profile and optimize performance-sensitive code. And also we have direct node.js bindings!
  • It is not doing a useless job under the hood. It is working directly with the low-level bytes buffer and avoids unnecessary memory allocations.
  • It is optimized by profiling produced assembly output πŸ‘―β€β™€οΈ

Check it out

Try it out right now! Give us your feedback, and do not forget about ⭐️ the project if you are interested!

GitHub logo dmtrKovalenko / odiff

The fastest pixel-by-pixel image visual difference tool in the world.

ODIFF

The fastest in the world pixel-by-pixel image difference tool.

made with reason npm

Why Odiff?

ODiff is a blazing fast native image comparison tool. Check benchmarks for the results, but it compares the visual difference between 2 images in milliseconds. It was originally designed to handle the "big" images. Thanks to Ocaml and its speedy and predictable compiler we can significantly speed up your CI pipeline.

Demo

base comparison diff
1diff
1diff
1diff

Features

  • βœ… .png, .jpg, .jpeg, .bmp – Files supported.
  • βœ… Cross-format comparison - Yes .jpg vs .png comparison is real.
  • βœ… Supports comparison of images with different layouts
  • βœ… Using YIQ NTSC transmission algorithm to determine visual difference
  • βœ… Zero dependencies for unix. Requires libpng only for windows

Coming in the nearest future:

  • ⏹ Ignoring regions
  • ⏹ Anti-aliasing support
  • ⏹ Remote images compare

Usage

Basic comparison

Run the simple comparison. Image paths can be one of supported formats, diff output can only…




Thank you for your time! Optimize your CI pipeline and in order to follow the tradition – no pixels were harmed in the making of this article πŸ™ˆ

Discussion (2)

pic
Editor guide
Collapse
maxziebell profile image
Max Ziebell

Great… looking forward to the Mac OS version and this would be a great tool for a SaaS undercutting current and expensive solutions by the gained performance and reduced demand on compute time.

Collapse
dmtrkovalenko profile image
Dmitriy Kovalenko Author

MacOS release is ready and out ✌️☺️. You can install it from binaries or npm. If u need some other package manager (brew is coming) β€” please let me a shout! Thanks!