TL;DR

I created an automation framework using Playwright, Python, and visual comparison, to test the creation of basic shapes in the AutoCAD Web application.

APP_USERNAME=ausername
APP_PASSWORD=apassword

pytest -v --headed --browser-channel chrome --base-url https://web.autocad.com -m regression

Here is a video of the automated test execution:

In this article, I'll go over how I approach the development of an end-to-end test automation framework from scratch, using AutoCAD Web as an example. Starting by analyzing the application under test, then selecting the most suitable tools, and ending with the development of the framework itself. While doing so, I'll demonstrate some of the capabilities of Playwright, how to use visual comparison for testing, and how to structure a UI end-to-end test framework.

About AutoCAD Web

Learning as much as possible about the application to be tested is the initial stage in an automation project. It is crucial to understand the context of the application in terms of features, business rules, and the full portfolio of the company. Then, the technical side should be considered, for example, what stack is used for development, the architecture, and services connected.

AutoCAD is a computer-aided design (CAD) software for drawing and modeling in 2D and 3D. Although it is mainly distributed as a desktop application, it also has a web version that allows editing, creating, and viewing CAD drawings in the browser. AutoCAD Web is only one of the products from the broad portfolio that Autodesk offers, and the one will focus on here.

These are the technical details of the application:

• It is a single-page application, developed with a modern front-end framework like React, Angular, or Vue. Based on some hints from the Chrome developer tools, I would say it is built with React.
• Some of the flows, for example, the login, require accessing separate windows.
• The drawing area is rendered on an HTML canvas element.
• Official documentation shows it is supported by Chrome, Firefox, Edge, and Safari on Windows and Mac.

Tests

For demonstration purposes, I'll be focusing on just a couple of test flows that will interact with the main views and validate core functionalities of the application.

These are the overall test steps:

1. Log in
2. Create a new drawing
3. Set color to a layer
4. Draw a circle or a rectangle
5. Change view
6. Verify that new drawings match reference:

Tooling selection

With a clear understanding of the details of the application, it is now possible to start exploring tools that will provide the suitable functionalities required to be able to automate the defined test flows.

Language: Python

It is assumed Python is the main language across the automation test team, and that there are other frameworks already developed on this language. Since Python is one of the most used languages by automation test developers and is supported by the majority of e2e automation test tools, it is definitively a good option. This would enable having the flexibility to quickly incorporate new developers into the project and also have a greater range of library options.

Alternatives:

• TypeScript: It is quickly becoming one of the most popular programming languages, due to its versatility to be used as a front and back-end language, with many options of development frameworks. For this specific project, it would also be a great alternative, mainly because Playwright with TS (node.js) offers an in-built capability for visual comparison and reporting, which is not available in other languages, plus taking advantage of the static-typed language benefits.

End-to-end test framework: Playwright

Playwright is an open-source web automation framework. It works particularly well with modern web apps(SPAs) since it has auto-wait capabilities by default. This means it waits for elements before actions are performed without the need to explicitly tell the program to wait until elements meet certain conditions, which is the traditional approach with frameworks like Selenium and the main cause of flakiness in test automation.

Furthermore, Playwright makes use of a WebSocket connection, rather than WebDriver API and HTTP, to interact with the browser. This results in faster execution since the entire execution happens on a single connection.

Both of these characteristics, in addition to its support for Chromium(and branded versions Chrome and Edge), Firefox, and WebKit(Safari), make Playwright a great fit for our goal. Be aware that, although Playwright can run on emulated tablet and mobile devices, it doesn't support automation on real devices.

Alternatives:

• Selenium: It could certainly get the job done since it is very robust and well-documented with the required capabilities for what is needed, however, dealing with explicit waits for elements and events would require some more work. Also, slower speed execution is a drawback.
• Cypress: Is also a very popular tool for modern web apps, that comes with automatic waiting and fast execution. However, some characteristics put Cypress down in the selection chart. It only supports JavaScript/Typescript, it does not support opening new tabs/windows natively, and the adoption can be challenging due to its unique architecture.

Visual comparison: pytest-playwrigh-visual

Before, during the AutoCAD Web research, it was indicated that the drawing area of the application is rendered as a canvas HTML element, which restricts the possibility of locating, selecting, and interacting with elements inside the canvas area. Without any more information or access to the core application code, we need to come up with a way to validate that whatever we draw using commands, is correctly displayed. For this, we use visual comparison, where a screenshot of the finished drawing is compared against a reference image, which represents the expected outcome.

Playwright with Python doesn't come with an in-built capability of visual comparison. Luckily, we can find pytest-playwrigh-visual, a Pytest plugin available in the PyPi repository. Although it is not a very rich plugin in terms of features, it does provide just what is required for our purpose, which is, besides doing the comparison, to create a snapshot on the first run(reference image), show the mismatches, and allow setting a comparison threshold.

The only concern I would have about pytest-playwrigh-visual is that the latest release was in April 2022, so it might become obsolete in the future with new Python or other dependencies versions. However, the implementation is very simple, so having to change to a different tool in the future would not represent a big waste.

Alternative

• Playwright in-built(only for node.js): This would be my first choice if TS/JS is being used since it comes with similar features as pytest-playwrigh-visual, with the advantage of being maintained directly by Playwright.
• Applitools: It is a more robust tool with a broad variety and sophisticated features, which comes with a monetary cost. So keep it in mind if the existing open-source options become a limitation and a budget is available.

Following table shows the final tooling selection:

The framework

Once all tools have been selected, it is time to put them together into a framework. The project structure and some other more specific artifacts, like environment handling, are also added here.

Playwright integrates gracefully with Pytest, in fact, it is the recommended test runner by Playwright using the official Playwright Pytest plugin. It comes with Playwright-specific fixtures for Pytest, where objects like Page and Context are initialized and easily passed to test methods.

With Pytest, test organization is intuitive and easily accomplished. Tests are organized in classes and methods, representing test suites and methods respectively. Pytest **markers **add an extra layer for test grouping, for example, marking tests as part of a smoke or regression execution. Finally, custom fixtures can be defined for set-up and tear-down procedures.

Project structure

The final framework structure is the following:

• autocad_app: contains all the page objects of the application. This means that each page or component in the application is represented as a class, elements as attributes, and actions as methods. This is the well-known POM pattern.

Below is the code for the Editor page editor_page.py as an example of a POM class:

class Editor:

    def __init__(self, page: Page):
        self.page = page
        self.properties_layer_color_dropdown = page.get_by_test_id('layer-color')
        self.command_line = page.locator('.class-cmdline-command-input')
        self.drawing_area = page.locator('div.input-adapter canvas')
        self.layout_tabs = page.locator('div.layout-tab-item')
        self.layout_tab_selected = page.locator('div.layout-tab-item.selected')

    def select_layer_color(self, layer_color: LayerColor):
        self.properties_layer_color_dropdown.click()
        self.page.get_by_test_id('menu-contents-container').get_by_text(layer_color.value).click()

    def create_circle_with_radius(self, center: str, radius: float):
        """Creates a circle from command line

        :param center: Example '0,0'
        :param radius: Example 350
        :return:
        """
        self.command_line.fill('CIRCLE')
        self.page.keyboard.press('Enter')
        self.page.get_by_text('Specify center point for circle or').wait_for()
        self.command_line.fill(center)
        self.page.keyboard.press('Enter')
        self.page.get_by_text('Specify radius of circle or').wait_for()
        self.command_line.fill(str(radius))
        self.page.keyboard.press('Enter')

• tests: contains test files and other required configuration files and test assets.

   -test_draw.py: contains test classes and methods

# Test suites can be represented by a class
class TestBasicDrawing:

    @pytest.mark.smoke  # Pytest markers are used to group test cases
    @pytest.mark.regression
    # Test cases can be represented by a method
    def test_draw_circles(self, logged_in_page: Page, assert_snapshot):
        page = logged_in_page
        Home(page).create_new_drawing()
        editor = Editor(page)
        editor.select_layer_color(Editor.LayerColor.RED)
        editor.set_grid("OFF")
        editor.create_circle_with_radius('300,300', 250)
        page.wait_for_timeout(4000)  # Waits for commands history to disappear
        editor.go_to_layout_tab('Layout1')
        assert_snapshot(editor.get_drawing_screenshot(), threshold=0.2)

   -conftest.py: contains custom fixtures accessible by the entire directory.

@pytest.fixture(name='logged_in_page')
def log_in(page: Page):
    """Goes through the login process

    :param page: Playwright Page object
    :return: a page after login
    """
    page.goto('/')
    Login(page).sign_in(USERNAME, PASSWORD)
    Home(page).wait_for()
    yield page

   -snapshots: contains reference images for visual comparison

• .env: contains environment variables, like db connection strings and credentials. Warning: This file should never be shared as it usually holds secret information.

APP_USERNAME=ausername
APP_PASSWORD=apassword

• pytest.ini: contains configurations for Pytest.

Test execution

Finally, once all parts are in place, we can now define how tests are going to be executed. Here it is important to consider the possible integration with a CI environment, which might require passing test arguments, like browser, tags, etc, in the execution command.

These are the available execution methods for the tests:

• IDE: Most popular IDE's are capable of discovering test methods from the code and provide graphical interfaces to trigger tests and visualize results. The following is an example with PyCharm, which was used for this project:

• Pytest CLI: Pytest enables execution from the command line using pytest command which also takes several flags to customize the execution. For example, the following command launches tests for the specified URL in headed mode for Chrome browser and only tests marked as regression:

pytest --base-url https://web.autocad.com --headed --browser-channel chrome -m regression

Run output looks like this:

Watch the test execution video in the TL;DR section at the top of the article.

Next steps

As part of the software development world, we should always be looking at continuously improving our products. So, although the resulting framework has a solid foundation to safely scale, there are still some missing key components that will increase its usefulness.

Reporting

The end goal of a test automation framework is to automatically validate what is working and what is not. There are several formats to visualize this outcome: plain text, XML, or HTML. The final selection should depend on who the report is intended for.

Some tools I would begin exploring are:
Pytest in-build reporting, pytest-html plugin and Allure Pytest library. As a side note, Playwright for node.js comes with reporting capabilities, which is another point in favor of using TS/JS.

CI/CD integration

We need to consider the DevOps environment that is already in place. It will be necessary to work in coordination with the DevOps team to be provided with the proper environment for the test execution, as well as the correct integration into the SDLC pipeline. Playwright has integrations already documented for the most popular CI/CD tools.

Behavior-Driven-Development(BDD) implementation:

When working tightly with the Product Management/BA team, and based on user stories, it might be of great benefit to implement a BDD testing approach. This way, acceptance criteria in user stories are easily translated into test steps in the framework using the Gherkin language (given-when-then), and have them nicely organized by features and scenarios. I would probably start by exploring Pytest-BDD library since Pytest is already integrated with Playwright. As a second option, I would explore Python's library behave.

Increase control of AutoCAD's drawing area.

Up to now, drawing in AutoCAD Web from the automation execution has been limited to actions through commands. However, there are at least 2 possible approaches we can take to gain more control over the drawing area so that we can increase test coverage:

• Playwright mouse actions: the mouse can be controlled by Playwright, allowing actions like moving the pointer and clicking with "left"|"right"|"middle" buttons. With this functionality, we can then emulate what is performed in the drawing area by a real user using the mouse. Although possible, this is a challenging approach, since the coordinates of every element must be known and there is no easy way to get feedback on the actions performed.
• Exposed drawing module(s): it is likely that the drawing is rendered into a canvas powered by one or more modules in the front-end code. Having access to those modules from the test code means attributes and methods could be used to interact with the drawing. To accomplish this, the required modules can be exposed to the DOM Document from the front-end code and then use JS executors in the automation framework to call the required methods. Be aware that this should only happen in a controlled QA environment.