Auto-Heal tests with recheck-web

This post is going to talk about auto-healing and the first thing that comes to your mind is Artificial Intelligence (AI). I do agree with that and it has been a cliche these days to make everything intelligent. The problem with AI is that you need to pre-process data, build a feature, pick an algorithm, and start training the system which may eventually take some time to improvise and get the desired outcome. Long story short, we don't have enough time for that in some cases. I am not offensive to AI, I evangelize about it all the time but still, it can't be applied for everything as "one shoe doesn't fit for all".

Auto-Healing

Auto-Healing in terms of computer science refers to the computer program that can correct itself whenever it encounters a blocker or a new change that has been incorporated in the system. In a normal scenario, the script would fail to proceed due to a lack of knowledge. Auto-healing makes sure that the system keeps up and running and does not break during its course.

The kind of auto-healing that we are going to see here is specific to UI test automation where most of the automation engineers would agree that most of their time has been spent on fixing the code due to constant changes in the front-end. This is prevalent now as we have element attributes in HTML that keep changing dynamically because of the recent front-end frameworks that are being used. To overcome this problem without further ado let me introduce a framework called recheck-web which is from a company called retest.

recheck-web

recheck-web works based on 'Difference Testing' where a Golden Master is created during the first run and it is considered as the reference. The Golden Master is a combination of screenshot and XML file which will have all the attributes related to elements on a web page. The following is a brief introduction to recheck-web from the company's official GitHub page.

You can explicitly or implicitly create Golden Masters (essentially a copy of the rendered website) and semantically compare against these. Irrelevant changes are easy to ignore and the Golden Masters are effortless to update. In case of critical changes that would otherwise break your tests, recheck-web can now peek into the Golden Master, find the element there, and (based on additional attributes) still identify the changed element on the current website.

In the above image, the overall concept has been simplified into a diagram. It is prudent that the UI is always subjected to changes, having that as a problem recheck-web does the following actions,

1. A usual selenium test will break if it finds a locator is changed in the UI and if we have not updated the test script and throws NoSuchElementException
2. A selenium test wrapped with recheck-web will have a Golden Master where all the attributes of the elements are captured for the entire page (only if you pass a driver but if a web element is passed, it would only capture the attributes of that particular element)
3. If a test script is run against the new UI whilst having the old locators in place, it would not break rather the test will execute as it would refer to the Golden Master and compares the present state of the locator with previous history a.k.a the Golden Master
4. The test would successfully execute all the functional validations but eventually will finish it with a warning and in a result, it shows what is expected and what has changed
5. This not only applies to the locator changes but also this validates the CSS and any Style changes as well which benefits the visual validation
6. The last step would be the decision making step where the user can accept all the changes or ignore it to update the Golden Master

It might seem to be complex to grasp but I will debrief it in a while with a working example.

Setup

1. You might need to ensure that Java and Maven are installed in your machine/server (Java 8 is the minimum version to be used)
2. Create a maven project and add the following dependencies to your pom.xml

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>5.6.2</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>de.retest</groupId>
            <artifactId>recheck-web</artifactId>
            <version>1.9.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>3.141.59</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>1.2.3</version>
            <scope>test</scope>
        </dependency>

The next thing you would need is an URL or a demo application against which you can test the script. I have created a sample login application which we will consider for this example.

Selenium Test

We can write a simple selenium test for the login app. Also, make a note that you should specify whichever driver you are using and the following driver should be made available to execute. In this case, I am using a Chrome driver.

import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import java.nio.file.Paths;

public class SeleniumTest {
    public WebDriver driver;

    @BeforeEach
    public void setupDriver() {
        System.setProperty("webdriver.chrome.driver","src/test/resources/drivers/chromedriver");
        driver = new ChromeDriver();
    }

    @Test
    public void breakableTest() throws Exception {
        String url = Paths.get("src/test/resources/loginpage.html").toUri().toURL().toString();
        driver.get(url);
        driver.findElement(By.id("username-field")).sendKeys("user");
        driver.findElement(By.id("password-field")).sendKeys("dev");
        driver.findElement(By.id("login-form-submit")).click();

        String result = driver.findElement(By.id("login-success-msg")).getText().equals("You have successfully logged in") ? "Login Pass" : "Login Fail";
        System.out.println(result);
    }

    @AfterEach
    public void windUp(){
        driver.quit();
    }
}

You could see from the above scenario that we need to explicitly make an assertion to find if the successful login message is displayed. Run this test and you will see that the test will pass. What if there are some UI changes that are made in the new release of your app. Consider the below example where you have the ID attributes 'username-field' and 'password-field'. It is now changed to 'username' and 'password' in the new release of the app.

<form id="login-form">
        <input type="text" name="username" id="username" class="login-form-field" placeholder="Username">
        <input type="password" name="password" id="password" class="login-form-field" placeholder="Password">
        <input type="submit" value="Login" id="login-form-submit">
</form>

But you could see that the selenium test script carries the id as 'username-field' and 'password-field' rather than 'username-field' and 'password-field'. This test would fail with NoSuchElementExeption.

org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"#username"}

Unbreakable Selenium Test

recheck-web truly stands to its name 'Unbreakable' where it creates a Golden Master with which it compares the current state. To create a recheck-web test, simply call the RecheckDriver and pass the driver argument to it as how you would do a RemoteWebDriver invocation. It is so simple and you don't need to add anything else. Of course, there are methods like 'startTest', 'cap', 'check, 'capTest' which can be called using RecheckImplementation class but you don't need to worry about that as all these are encompassed in RecheckDriver class.

import de.retest.web.selenium.RecheckDriver;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.chrome.ChromeDriver;

import java.nio.file.Paths;
public class UnbreakableSeleniumTest {
    RecheckDriver driver;

    @BeforeEach
    public void setupRecheck() {
        System.setProperty("webdriver.chrome.driver","src/test/resources/drivers/chromedriver");
        driver = new RecheckDriver(new ChromeDriver());
    }

    @Test
    public void unBreakableTest() throws Exception {
        String url = Paths.get("src/test/resources/loginpage.html").toUri().toURL().toString();
        driver.get(url);
        driver.findElement(By.id("username-field")).sendKeys("user");
        driver.findElement(By.id("password-field")).sendKeys("dev");
        driver.findElement(By.id("login-form-submit")).click();
    }

    @AfterEach
    public void windUp(){
        driver.quit();
    }
}

Consider the below login HTML.

<form id="login-form">
        <input type="text" name="username" id="username-field" class="login-form-field" placeholder="Username">
        <input type="password" name="password" id="password-field" class="login-form-field" placeholder="Password">
        <input type="submit" value="Login" id="login-form-submit">
</form>

Now if you run the test it will fail for the first time as you don't have a Golden Master created yet.

Run the test again and you will see that the test will succeed but with some errors as mentioned below in the picture.

This assures that our present state is compared against the Golden Master. The errors that you see are some metadata and attributes that are compared against the Golden Master which can be ignored by writing them in the recheck.ignore file generated in .retest folder. Add attribute=grid-template-rows and metadata=window.height in the recheck.ignore file.

attribute=grid-template-rows
metadata=window.height

Re-run the test and it will succeed without any errors.

Let us now change the login HTML and tweak the ID to 'username' and 'password'.

<form id="login-form">
        <input type="text" name="username" id="username" class="login-form-field" placeholder="Username">
        <input type="password" name="password" id="password" class="login-form-field" placeholder="Password">
        <input type="submit" value="Login" id="login-form-submit">
</form>

If you run the test against this new change, the execution will not break but it will refer to the Golden Master that is available since the first run and it will compare the similar attributes that were present earlier and make a one-to-one connection, thereby finds the new element without even throwing any exception. Obviously, the test will report with errors (i.e. the locator change) that were found during the execution.

Now you can see the difference that the expected was 'username-field' but the actual ID attribute was 'username'. The same applies to the ID attribute 'password'. It is not a rule that Golden Master always refers to the outcome that was derived from the first run. We need to update the Golden Master to the latest stable UI changes by liaising with the development community. So we could clearly see that there are often changes happening in and out of the script. This might look like Déjà vu to you as recheck-web has a CLI called recheck.cli which is similar to the operations of GIT. It is imperative that every time you cannot manually update the recheck.ignore file to filter the result and to commit or accept the new UI changes to the Golden Master. This is where recheck.cli would help us in speeding up the process.

recheck.cli

In order to install and setup the CLI, refer to this documentation. To ensure if the recheck is working, restart your CMD/Terminal after the installation and type recheck. It should return with list of options that are available.

$ recheck

Usage: recheck [--help] [--version] [COMMAND]
Command-line interface for recheck.
      --help      Display this help message.
      --version   Display version info.
Commands:
  account     Allows to log into and out of your account and show your API key.
  help        Displays help information about the specified command
  commit      Accept specified differences of given test report.
  completion  Generate and display an auto completion script.
  diff        Compare two Golden Masters.
  ignore      Ignore specified differences of given test report.
  show        Display differences of given test report.
  version     Display version info.

Every recheck test will generate a report in target/test-classes/retest/recheck/YOURTESTNAME.report. You could run this report using the recheck.cli by hitting the following command by navigating to the report folder.

$ recheck show YOURTESTNAME.report

To accept all the recent changes in your test related to ID attributes, you can simply commit all at once by issuing the below command,

$ recheck commit --all YOURTESTNAME.report

If you go back and check your retest.xml the attributes would have changed from 'username-field' to 'username' and 'password-field' to 'password'. Now that you have updated the Golden Master, you have to update the corresponding test scripts as well else the test would fail because we don't have a history of 'username-field' or 'password-field' anymore. In order to auto-heal, retest has a GUI called review which is an insightful dashboard that can automatically make the relevant changes in Golden Master and in the test script.

Also, recheck-web has an important feature called virtual ID. It creates the virtual ID for all the states viz. username, password, and login button. If you want your test to be completely unbreakable, using virtual IDs will make your life worry-free. The retestId has to be imported from the package de.retest.web.selenium.By.

import de.retest.web.selenium.RecheckDriver;
import de.retest.web.selenium.By;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.openqa.selenium.chrome.ChromeDriver;
import java.nio.file.Paths;

public class UnbreakableSeleniumTest {
    RecheckDriver driver;

    @BeforeEach
    public void setupRecheck() {
        System.setProperty("webdriver.chrome.driver","src/test/resources/drivers/chromedriver");
        driver = new RecheckDriver(new ChromeDriver());
    }

    @Test
    public void unBreakableTest() throws Exception {
        recheck.startTest();
        String url = Paths.get("src/test/resources/loginpage.html").toUri().toURL().toString();
        driver.get(url);
        driver.findElement(By.retestId("usernamefield")).sendKeys("user");
        driver.findElement(By.retestId("passwordfield")).sendKeys("dev");
        driver.findElement(By.retestId("loginformsubmit")).click();
    }

    @AfterEach
    public void windUp(){
        driver.quit();
    }
}

Check the recheck-web documentation for more insights and you can also pull my sample implementation from GitHub.