Visual Automation with Applitools

The purpose of this article is to review how we can integrate Applitools platform to add AI/ML in our existing Web UI Functional Tests.

Applitools Introduction

Applitools is "yet" another tool that you can use in conjunction with Selenium to handle the complicated visual types of testing you need to get done, and leave the more functionality-type testing to the tools that where designed to handle that, like Selenium.

Applitools provide an easy to use platform to review visual differences between screens that can handle different configurations we will examine later on (e.g view port, device types etc.). It provides integration with multiple test frameworks and programming languages in order for any team to be able to adjust to their needs.

Applitools Integration with Test Frameworks

There are a bunch of SDKs for most of the programming languages that allow you to take images and upload them for visual validation in Applitools. We are going to review on how to start with the integration in a Java based Test Project.

Dependencies and SDKs

In your maven dependency management section add the following dependencies:

 <!-- START Applitools Eyes dependencies -->
   <dependency>
       <groupId>com.applitools</groupId>
       <artifactId>eyes-selenium-java3</artifactId>
       <version>${eyes-java3.version}</version>
   </dependency>
   <dependency>
       <groupId>com.applitools</groupId>
       <artifactId>eyes-images-java3</artifactId>
       <version>${eyes-java3.version}</version>
    </dependency>
 <!-- END Applitools Eyes dependencies -->

Before any selenium test starts execution we need to initialize Eyes and EyesRunner objects in order to take images within our tests.

An example of initializing the runner:

 EyesRunner runner;
        if (visualGrid) {
            LOG.info("Applitools: Running on Visual Grid");
            runner = new VisualGridRunner(concurrency);
        } else {
            LOG.info("Applitools: Running on Classic Runner");
            runner = new ClassicRunner();
        }
        Configuration config = new Configuration();
        config.addBrowser(1920, 1080, BrowserType.CHROME)
                .addBrowser(1920, 1080, BrowserType.FIREFOX)
                .addDeviceEmulation(DeviceName.iPhone_X, ScreenOrientation.PORTRAIT)
                .setApiKey(apiKey)
                .setBatch(batch)
                .setMatchLevel(!matchLevel.isEmpty() ? MatchLevel.valueOf(matchLevel) : MatchLevel.LAYOUT);

Lets review the arguments of the above example. First we need to set the type of Runner and currently Applitools support two flavors:

• Classic Runner: Applitools will process the images taken in your local executed browser/device
• Visual Grid Runner: Applitools give you the ability to send images and process them in different type of configurations (browsers/devices) scaling your tests on different platforms covering more visual checkpoints Next we have the API Key which is provided by an Admin user of the platform. Batch can be used to set the batch name that will be used to store the tests for this runner and finally the Match Level which is the type of validations that will be used to compare the checkpoints.

Applitools Eyes can test the UI in 4 different comparison levels:

Exact (MatchLevel.EXACT) - pixel-to-pixel comparison, not recommended.

Strict (MatchLevel.STRICT) - Strict compares everything including content (text), fonts, layout, colors and position of each of the elements. Strict knows to ignore rendering changes that are not visible to the human. Strict is the recommended match level when running regression tests on the same browser/OS (strict is not designed to do cross browser comparison).

Content (MatchLevel.CONTENT) - Content works in a similar way to Strict except for the fact that it ignores colors.

Layout (MatchLevel.LAYOUT) - Layout, as its name implies, compares the layouts (i.e. structure) of the baseline and actual images. It validates the alignment and relative position of all elements on the page, such as: buttons, menus, text areas, paragraphs, images, and columns. It ignores the content, colour and other style changes between the pages.

Now lets see how to initialize the Eyes instance

Eyes eyes = new Eyes(runner);
eyes.setConfiguration(config);
eyes.setForceFullPageScreenshot(true);
eyes.setStitchMode(StitchMode.CSS);
eyes.setWaitBeforeScreenshots(2000);
eyes.setLogHandler(new FileLogger("target/eyes.log", true, true));

Now that we initialized our eyes before our test we can actually navigate to a page with selenium and take a baseline image from the page and send it over to Applitools.

log.info("Start Visual Check From Applitools");
int matchTimeout = 10000;// in milli seconds
if (!eyes.getIsOpen()) {
    eyes.open(driver, "VisualTest","TestName");
}
    eyes.checkWindow(matchTimeout, "ScreenshotName");
}

With the commands above we create a new test and send an image of the page into Applitools for visual comparison.

Getting Results from Applitools

First we need to close eyes instance after a test is finished execution with

boolean testFailed = result.getStatus() == ITestResult.FAILURE;
if (!testFailed) {
      eyes.closeAsync();
     } else {
      eyes.abortAsync();
     }
}

This terminates the sequence of checkpoints, and then waits asynchronously for the test results.

We can retrieve the results now by calling:

TestResultsSummary allTestResults = runner.getAllTestResults(false);
for (TestResultContainer result1 : allTestResults) {
    Throwable ex = result1.getException();
    TestResults result = result1.getTestResults();
    if (result == null) {
    LOG.info("No test results information available\n");
    } else {
String resultReport = String.format("URL = %s, AppName = %s, testname = %s, Browser = %s,OS = %s, viewport = %dx%d, matched = %d,mismatched = %d, missing = %d,aborted = %s\n", result.getUrl(),
 result.getAppName(),
 result.getName(),
 result.getHostApp(),
 result.getHostOS(),
 result.getHostDisplaySize().getWidth(),
 result.getHostDisplaySize().getHeight(),
 result.getMatches(),
 result.getMismatches(),
 result.getMissing(),
 (result.isAborted() ? "aborted" : "no"));
 LOG.info("Applitools Results: " + resultReport);
}

So with this way we can get results and post them to our test reporting platforms.

Applitools CI/CD Integration

In order to view test results directly in your Jenkins CI server you need first to install Applitools Jenkins plugin.

Then we need to add the following section in your jenkins pipeline stage for running the visual tests:

node {
     stage('Visual Tests') { 
         Applitools() {
               sh 'mvn clean test'
         }
     }
}

After that the results are available in an embedded iframe in Jenkins: