In the previous article, we talked about WebdriverIO. A tool that has become popular very fast in the field of test automation due to its powerful advantages. In this article, we will talk about one of its advantages that has advanced our test automation suite. We have an application that helps our clients develop stunning websites. Whenever a new feature is developed that may impact the UI of the site, WebdriverIO can help us test the change in the UI of the sites. WebdriverIO provides a service called as wdio-image-comparison-service, which performs pixel-by-pixel image comparison.
We already have a wdio-based automation project that does end-to-end functional testing. We expanded it to screenshot test all the pages for thousands of sites. We explored various tools for image comparison. We then discovered that WebdriverIO furthermore offers an image comparison tool. Then we began looking into how we could incorporate this "wdio-image-comparison-service" into our already-running automation project.
Set-up
Referring to the official doc for wdio-image-comparison-service
, we installed the module:
npm install --save wdio-image-comparison-service
We added the below options to the Test runner services in wdio.conf.js
file:
services: [
'chromedriver',
[
'image-comparison',
// The options
{
// Some options, see the docs for more
baselineFolder: join(process.cwd(), './tests/'),
formatImageName: '{tag}-{logName}-{width}x{height}',
screenshotPath: join(process.cwd(), '.tmp/'),
savePerInstance: true,
autoSaveBaseline: true,
blockOutStatusBar: true,
blockOutToolBar: true,
disableCSSAnimation: false,
ignoreColors: true,
ignoreTransparentPixel: true,
},
],
],
Above is the default setup. With this default setup, we thought that we had completed configuring the setup and the next step was to create a spec file to compare screenshots.
-
browser.saveScreen("homepage")
function saves the screenshot as a baseline image with the namehomepage.png
. -
browser.checkScreen("homepage")
function saves the screenshot of the current page and compares it with the baseline image having the same name i.e.homepage.png
.
However, we encountered an issue with the default setup. When we attempted to create a spec to compare screenshots of the two different doodles, it did not detect any mismatches, and the spec passed. We will explain this issue in more detail below:
We created a spec file called image_compare.js
as follows:
describe("Image Comparison", () => {
it("saves the baseline image", async () => {
await browser.url("https://www.google.com/doodles/teachers-day-2021-september-11");
await browser.saveScreen("homepage");
});
it("compares with baseline image", async () => {
await browser.url("https://www.google.com/doodles/teachers-day-2022-september-11");
const result = await browser.checkScreen("homepage");
expect(result).toEqual(0);
});
});
In the above spec, we were comparing screenshots of the Teacher's Day - 2021
homepage and Teacher's Day - 2022
homepage. And we were expecting some mismatch percentage. But the spec got passed. When we investigated, we found that wdio-image-comparison-service
saved the baseline image under .tmp
folder. The .tmp
folder has actual
as a subfolder. Under the actual
folder, it saved the baseline image.
In our case, when we ran the spec, it overrode Teacher's Day - 2021 homepage.png
with Teacher's Day - 2022 homepage.png
inside .tmp/
folder. Inside ./tests/
folder also, it saved Teacher's Day - 2022 homepage.png
. When we investigated more on this, then we got to know that internally this package uses fs-extra
plugin and it has limited Windows support. But, running the same script on Mac or Linux did not result in any issues or failures. We also found out that this is an existing issue. We have also added a comment there about what could be a workaround.
Workaround
As we previously said, the baseline picture is saved by 'wdio-image-comparison-service' by default under the '.tmp' folder. In 'wdio.conf.js', we removed the following code from the'services' section:
baselineFolder: join(process.cwd(), './tests/'),
screenshotPath: join(process.cwd(), '.tmp/'),
Then, we specified the testOptions
in the it
block where screenshots are compared.
const testOptions = {
actualFolder: join(process.cwd(), './.tmp/checkActual'),
baselineFolder: join(process.cwd(), './.tmp/actual'),
diffFolder: join(process.cwd(), './.tmp/testDiff'),
returnAllCompareData: true,
};
-
checkActual
: The current image will be saved in this folder. It is called asactualFolder
. -
actual
: The baseline image will be saved in this folder. It is called asbaselineFolder
. -
diffFolder
: This image gets created with all the mismatches highlighted after the two screenshots are compared. -
returnAllCompareData
: By default,check
methods of this service returns only mismatch percentage. When this option is passed astrue
, then it returns the current imagefilename
,folders
that contain the path ofactual
,baseline
, anddiff
, andmisMatchPercentage
.
After these modifications, the updated code is as follows:
wdio.conf.js
services: [
'chromedriver',
[
'image-comparison',
// The options
{
// Some options, see the docs for more
savePerInstance: true,
autoSaveBaseline: true,
blockOutStatusBar: true,
blockOutToolBar: true,
disableCSSAnimation: false,
ignoreColors: true,
ignoreTransparentPixel: true,
},
],
],
image_compare.js
import { join } from 'path';
describe("Image Comparison", () => {
it("saves the baseline image", async () => {
await browser.url("https://www.google.com/doodles/teachers-day-2021-september-11");
await browser.saveScreen("homepage");
});
it("compares with baseline image", async () => {
await browser.url("https://www.google.com/doodles/teachers-day-2022-september-11");
const testOptions = {
actualFolder: join(process.cwd(), './.tmp/checkActual'),
baselineFolder: join(process.cwd(), './.tmp/actual'),
diffFolder: join(process.cwd(), './.tmp/testDiff'),
returnAllCompareData: true,
};
const result = await browser.checkScreen("homepage", testOptions);
expect(result).toEqual(0);
});
});
When we passed testOptions
as a parameter to the checkScreen
function, then wdio
easily identified where our baseline images are saved and with which image to compare.
Now, if you run the above spec, then it will properly do the comparison, highlight the mismatch, and return the mismatch percentage.
Teacher's Day - 2021 | Teacher's Day - 2022 |
Image Difference | |
The wdio-image-comparison-service
is not just limited to capturing and comparing screens, it can also compare the full page screen, and elements, and also checks if the website supports the keyboard's TAB
key. Please refer to wdio-image-comparison-service
options.
Mobile Screenshot Testing
If screenshot testing for mobile devices is needed, then just resizing your browser is never a good idea because this module compares visuals of what the end user is going to see. Wdio Devtools can be helpful in such cases. Wdio Devtools
is in itself a distinct, extensive subject that offers additional benefits.
We just need to pass service
as devtools
in wdio.conf.js
file. We can emulate a specific device by await browser.emulateDevice('iPhone X')
. Please refer to Device emulation.
We have created a small demo project wdio-image-comparison-demo that you can clone for better understanding. You can explore other options available for wdio-image-comparison-service
.
References:
Top comments (0)