Tools4AI with Selenium can revolutionize UI validation by automating the process of verifying UI elements and ensuring consistency with design specifications. This approach goes beyond traditional UI validation methods by validating entire UI sections at once instead of examining individual elements. As a result, Tools4AI integration with Selenium can significantly streamline the testing process, allowing you to automate a comprehensive verification of web-based applications.
With this integration, you can leverage a combination of natural language and Java code to create Selenium test scripts in a more human-readable format. This simplifies UI testing and increases efficiency by allowing non-programmers to write test scenarios in plain English.
Selenium Integration with Tools4AI
Tools4AI's integration with Selenium introduces a flexible way to automate UI testing. Instead of traditional Java code for Selenium scripts, Tools4AI allows you to define test scenarios in plain English, offering a more accessible approach to testing web applications. These English-based commands can be converted into Selenium code to automate web-based interactions and streamline testing.
Example of Selenium Test with Tools4AI
WebDriver driver = new ChromeDriver();
SeleniumProcessor processor = new SeleniumGeminiProcessor(driver);
// Navigate to the website
processor.processWebAction("go to website https://the-internet.herokuapp.com");
// Check if a specific button is present
boolean buttonPresent = processor.trueFalseQuery("do you see Add/Remove Elements?");
if (buttonPresent) {
// Perform a click action
processor.processWebAction("click on Add/Remove Elements");
// Further English-based instructions can be added
}
// Check if checkboxes are visible and interact with them
processor.processWebAction("go to website https://the-internet.herokuapp.com");
boolean isCheckboxPresent = processor.trueFalseQuery("do you see Checkboxes?");
if (isCheckboxPresent) {
processor.processWebAction("click on Checkboxes");
processor.processWebAction("select checkbox 1");
}
In this example, the SeleniumProcessor processes commands in plain English and converts them into Selenium actions.
Code for above example is here
This approach allows for complex interactions without manually writing Java code for each test. Tools4AI serves as a bridge between natural language and Selenium, making it easier to automate UI testing in a way that is both efficient and intuitive.
You can use SeleniumOpenAIProcessor as well but it will use plain html conversion to Pojo rather than images as OpenAI does not support API for images yet
Screen Validations
One of the most powerful and unique features of Tools4AI is its ability to convert an image or an html page into a structured Java object. This capability can be invaluable for various applications, especially in scenarios where you need to extract data from an image, screenshot, html page, then manipulate or analyze it within your Java code.
By transforming an image into a Java object, Tools4AI opens up a wide range of possibilities:
UI Testing and Validation: You can convert screenshots of a user interface into Java objects to validate UI elements and their attributes. This feature simplifies the testing process by allowing you to automate the verification of entire sections or components without manually interacting with individual elements.
Data Extraction: Tools4AI's image-to-POJO functionality can be used to extract data from images, such as scanned documents, infographics, or screenshots, and convert it into a structured format. This can be useful for creating data-driven applications, automating workflows, or extracting information for further processing.
Simplifying Automated Tests: By converting images into Java objects, you can write automated tests that operate at a higher level of abstraction. Instead of interacting with individual web elements, you can work with an entire data structure that represents the content of a web page or UI screen.
Examples
Example 1
The above picture can be converted to Java object with simple code in this way
GeminiImageActionProcessor processor = new GeminiImageActionProcessor();
Sales sales = (Sales)processor.imageToPojo(GeminiImageExample.class.getClassLoader().getResource("images/sales.PNG"), Sales.class)
The Sales pojo looks like this
@Getter
@Setter
@ToString
@NoArgsConstructor
public class Sales {
@MapKeyType(Integer.class)
@MapValueType(Double.class)
Map<Integer,Double> yearlySales;
}
If you dont want to create a Pojo you can get data in simple HashMap or Json String as well
log.info(processor.imageToJson(GeminiImageExample.class.getClassLoader().getResource("images/sales.PNG"),"sales in 2013"));
and the response would be
INFO: {"fields":[{"fieldName":"sales in 2013","fieldType":"String","fieldValue":"58"}]}
Or you can get multiple values in this way
log.info(processor.imageToJson(GeminiImageExample.class.getClassLoader().getResource("images/sales.PNG"),"sales in 2013", "sales in 2015"));
and your output will be
INFO: {
"fields": [
{
"fieldName": "sales in 2013",
"fieldType": "String",
"fieldValue": "58"
},
{
"fieldName": "sales in 2015",
"fieldType": "String",
"fieldValue": "67"
}
]
}
Example 2
We will convert this Pojo into Java code
FoodConsumption foodConsume = (FoodConsumption) processor.imageToPojo(GeminiImageExample.class.getClassLoader().getResource("images/PieChart.PNG"), FoodConsumption.class);
log.info(foodConsume.toString());
FoodConsumption Pojo looks like this
@Getter
@Setter
@NoArgsConstructor
@ToString
public class FoodConsumption {
@MapValueType(Double.class)
@MapKeyType(String.class)
private Map<String, Double> foodTypeToPercentage;
}
Output looks like this
INFO: FoodConsumption(foodTypeToPercentage={Rice Dishes=0.3, Leafy Greens=0.15, Soups=0.25, Root Vegetables=0.2, Hot Drinks=0.1})
Example 3
log.info(processor.imageToPojo(GeminiImageExample.class.getClassLoader().getResource("images/FruitsSold.PNG"), WeeklyFruitSales.class).toString());
and our pojo looks like this
@Getter
@Setter
@NoArgsConstructor
@ToString
public class WeeklyFruitSales {
@ListType(DailyFruitSales.class)
private List<DailyFruitSales> dailySales;
// Constructor
public WeeklyFruitSales(List<DailyFruitSales> dailySales) {
this.dailySales = dailySales;
}
// Getter and Setter
public List<DailyFruitSales> getDailySales() {
return dailySales;
}
public void setDailySales(List<DailyFruitSales> dailySales) {
this.dailySales = dailySales;
}
}
and output is
INFO: WeeklyFruitSales(dailySales=[DailyFruitSales(dayOfWeek=Monday, fruitSales={Mango=12, Orange=10, Banana=5}), DailyFruitSales(dayOfWeek=Tuesday, fruitSales={Mango=15, Orange=13, Banana=6}), DailyFruitSales(dayOfWeek=Wednesday, fruitSales={Mango=7, Orange=9, Banana=6}), DailyFruitSales(dayOfWeek=Thursday, fruitSales={Mango=6, Orange=14, Banana=5}), DailyFruitSales(dayOfWeek=Friday, fruitSales={Mango=19, Orange=17, Banana=8}), DailyFruitSales(dayOfWeek=Saturday, fruitSales={Mango=19, Orange=21, Banana=10}), DailyFruitSales(dayOfWeek=Sunday, fruitSales={Mango=15, Orange=21, Banana=9})])
Example 4
Imagine you're testing an online library system, and you encounter a complex user interface with various elements representing different sections, like books, members, and other UI components. Traditionally, you would inspect each element individually to validate its content and functionality. This involves identifying and interacting with each UI element separately, which can be time-consuming and error-prone.
Tools4AI transforms this process by offering a unique feature that converts an entire screen or webpage into a Plain Old Java Object (POJO). This powerful capability enables you to extract the structure and content of a complex UI in one step, significantly streamlining the testing process
The POJOS look like this
@Setter
@Getter
@NoArgsConstructor
@ToString
public class LibraryScreen {
@ListType(Book.class)
private List<Book> latestBooks;
@ListType(Member.class)
private List<Member> members;
}
and
@Setter
@Getter
@NoArgsConstructor
@ToString
public class Book {
private String title;
private String author;
private String genre;
private boolean isAvailable;
// Constructors, getters, and setters
}
and
@Setter
@Getter
@NoArgsConstructor
@ToString
public class Member {
private String id;
private String name;
@Prompt(dateFormat = "ddMMyyyy")
private Date membershipStart;
private int booksLoaned;
// Constructors, getters, and setters
}
Please pay special attention to @Prompt(dateFormat = "ddMMyyyy") as this will convert the date to the specified format automatically
Since this is a complex screen we can use Transformer
String text = processor.imageToText(GeminiImageExample.class.getClassLoader().getResource("images/library.PNG"),"convert the entire screen to text");
GeminiV2PromptTransformer transformer = new GeminiV2PromptTransformer();
log.info(transformer.transformIntoPojo(text, LibraryScreen.class).toString());
and the entire screen will be converted to POJO
INFO: LibraryScreen(latestBooks=[Book(title=The Great Gatsby, author=F. Scott Fitzgerald, genre=Fiction, isAvailable=true), Book(title=1984, author=George Orwell, genre=Dystopian, isAvailable=false)], members=[Member(id=001, name=Alice Smith, membershipStart=Thu Jan 12 00:00:00 EST 2023, booksLoaned=4), Member(id=002, name=Bob Johnson, membershipStart=Wed Feb 15 00:00:00 EST 2023, booksLoaned=2)])
Example 5
String jsonStr = processor.imageToJson(GeminiImageExample.class.getClassLoader().getResource("images/auto.PNG"),"Full Inspection");
log.info(jsonStr);
and the result is
INFO: {
"fields": [
{
"fieldName": "Full Inspection",
"fieldType": "String",
"fieldValue": "Starting at $99.99"
}
]
}
Or it can be converted into Pojo like this
@Getter
@Setter
@ToString
@NoArgsConstructor
@AllArgsConstructor
public class AutoRepairScreen {
double fullInspectionValue;
double tireRotationValue;
double oilChangeValue;
Integer phoneNumber;
String email;
String[] customerReviews;
}
Example 6
Imagine encountering the project report screen shown in the image you uploaded. Tools4AI can transform this screen into a Java object, allowing you to interact with its data in a structured manner for further analysis or testing.
We will convert this entire project report to POJO so that we can take action
@NoArgsConstructor
@AllArgsConstructor
@Getter
@Setter
@EqualsAndHashCode
@ToString
public class ProjectDashboard {
@MapKeyType(String.class)
@MapValueType(Double.class)
private Map<String, Double> featuresImplemented; // Map of quarter to percentage
@MapKeyType(String.class)
@MapValueType(Double.class)
private Map<String, Double> expenses; // Map of quarter to percentage
@ListType(ProjectStatus.class)
private List<ProjectStatus> projectStatuses; // List of project status entries
@ListType(Task.class)
private List<Task> tasks; // List of tasks
@ListType(String.class)
private List<String> criticalItems; // List of critical items
@ListType(String.class)
private List<String> blockers;
}
and
@NoArgsConstructor
@AllArgsConstructor
@Getter
@Setter
@EqualsAndHashCode
@ToString
public class Task {
private String assignedTo;
private String priority;
private String status;
private int completion;
}
and
@NoArgsConstructor
@AllArgsConstructor
@Getter
@Setter
@EqualsAndHashCode
@ToString
public class ProjectStatus {
private String projectName;
@MapKeyType(String.class)
@MapValueType(Integer.class)
private Map<String, Integer> statusCounts;
}
and then we call our image processor
GeminiImageActionProcessor processor = new GeminiImageActionProcessor();
ProjectDashboard projectDashboard = (ProjectDashboard) processor.imageToPojo(GeminiImageExample.class.getClassLoader().getResource("images/RAG.PNG"), ProjectDashboard.class);
log.info(projectDashboard.toString());
and here is the result
INFO: ProjectDashboard(featuresImplemented={1st Qtr=64.0, 3rd Qtr=11.0, 2nd Qtr=25.0}, expenses={1st Qtr=25.0, 3rd Qtr=40.0, 2nd Qtr=35.0}, projectStatuses=[ProjectStatus(projectName=Neo, statusCounts={Issues=3, Features=2, Backlog=4}), ProjectStatus(projectName=Wypal, statusCounts={Issues=3, Features=1, Backlog=2}), ProjectStatus(projectName=Dorake, statusCounts={Issues=1, Features=3, Backlog=2}), ProjectStatus(projectName=Symphony, statusCounts={Issues=2, Features=3, Backlog=1})], tasks=[Task(assignedTo=John, priority=High, status=Done, completion=100), Task(assignedTo=Smith, priority=Normal, status=In progress, completion=20), Task(assignedTo=Zoya, priority=Low, status=Not started, completion=0), Task(assignedTo=Ellie, priority=High, status=In progress, completion=40)], criticalItems=[Order more RAM], blockers=[Server Upgrades, Core Processors])
Conclusion
Tools4AI's image-to-Java object conversion feature provides a bridge between raw visual data and structured information, allowing you to process, analyze, and validate data in a way that is both efficient and accessible. It can streamline many tasks, from automated UI testing to data-driven applications, by enabling complex data manipulations with minimal effort.
Code for this article is here
๐ฅ ๐๐๐ฏ๐จ๐ฅ๐ฎ๐ญ๐ข๐จ๐ง๐ข๐ณ๐ ๐๐จ๐ฎ๐ซ ๐๐ ๐๐๐ฌ๐ญ๐ข๐ง๐ ๐ฐ๐ข๐ญ๐ก ๐๐จ๐จ๐ฅ๐ฌ4๐๐! ๐ฅ
Tired of manually inspecting every element in a complex UI? Tools4AI is here to change the game. This groundbreaking technology converts entire screens or images into structured Java objects (POJOs), enabling you to streamline your automated UI testing, data extraction, and data analysis. No more tedious element-by-element validationโTools4AI does it all in one step!
๐๐๐ฒ ๐๐ข๐ ๐ก๐ฅ๐ข๐ ๐ก๐ญ๐ฌ:
๐๐๐จ๐ฆ๐ฉ๐ซ๐๐ก๐๐ง๐ฌ๐ข๐ฏ๐ ๐๐๐ญ๐ ๐๐ฑ๐ญ๐ซ๐๐๐ญ๐ข๐จ๐ง: Convert entire screens into Java objects for efficient data processing.
โก ๐๐ข๐ฆ๐ฉ๐ฅ๐ข๐๐ข๐๐ ๐๐ฎ๐ญ๐จ๐ฆ๐๐ญ๐ข๐จ๐ง: Automate UI testing with a structured approach, interacting with POJOs instead of individual elements.
๐ฅ ๐๐ง๐ก๐๐ง๐๐๐ ๐๐๐๐ข๐๐ข๐๐ง๐๐ฒ: Quickly validate entire UI sections to ensure design consistency and accuracy.
๐๐ฌ๐๐ฌ:
๐ฅ ๐๐๐๐ฅ๐ญ๐ก๐๐๐ซ๐: Convert images of medical records into structured data to streamline electronic health records (EHR) management and automate data entry.
๐ ๐๐๐ญ๐๐ข๐ฅ: Turn product images into Java objects to enhance inventory management and automate real-time stock tracking.
๐ซ ๐๐๐ฎ๐๐๐ญ๐ข๐จ๐ง: Convert classroom whiteboard notes or scanned documents into Java objects for automated digitization and learning analytics.
๐ญ ๐๐๐ง๐ฎ๐๐๐๐ญ๐ฎ๐ซ๐ข๐ง๐ : Automate quality control by converting product images into Java objects to detect defects and ensure compliance with manufacturing standards.
๐ฐ ๐
๐ข๐ง๐๐ง๐๐: Transform financial reports or stock data into Java objects to automate financial analysis and streamline data processing.
Top comments (0)