From Instability to Reliability: Using Puppeteer and Bubanai for e2e stabilization

Dear Community

I worked as QA Automation / SDET at WiX and I want to share my experience using Puppeteer and stabilising e2e tests in TypeScript.

My last project was EditorX - a platform for creating responsive websites using drag & drop technology. Over the years, we've written a significant number of e2e tests (2000+) in TypeScript, but we consistently faced stability issues in our e2e tests due to most user interaction logic being executed on the frontend, and the consequences of actions may occur some time after the action actually was performed.

Puppeteer, in general, executes it's functions much faster than the system reacts to commands. Initially, we used basic Puppeteer methods and native Jest assertions, but over time, we realised that we needed something more to achieve build stability.

What we decided to change

We have implemented the Continuous Integration process on our project. The entire test suite runs on a ready pull request, and the code can only be merged into the master branch after passing all tests. The requirement for success rate (SR) is not less than 99.6% (excluding failures caused by real bugs or infrastructure issues). If a test does not meet the stability criteria, it is skipped. To address the SR issues and improve the stability of our e2e tests, we developed our own utilities within our project and later noticed that other teams were facing similar stability issues that we had in the past.

So, we decided to extract them into an open-source library called Bubanai.

Bubanai translates from Hebrew as a puppeteer.

Features provided by Bubanai

This open-source library provides the following capabilities:

• Simplified Puppeteer methods by adding expectations inside common methods like click and hover, etc.
• Fast access to element attributes/properties.
• Expectations for elements, for example, waiting for an element to have a certain attribute or part of it; waiting for an element's position not to change, etc.
• Expectations for executing functions based on given conditions.
• Context handling methods.
• Collection handling methods.
• Simplified drag & drop.
• Logging for high-level test methods.
• Scroll utilities.Keyboard interactions.
• Console & Network listeners.
• Utilities for initialized nested elements (useful for early versions of Puppeteer).
• Utilities for dropdowns.
• etc.

Typical problems this library solves:

Example 1

Let's say you have a component that changes its color after a certain time delay following a click action. You know exactly the color you're setting it to, but after the change, it does not immediately take on the desired state.

import { getElementBackgroundColor, waitForObjectsToBeEqual } from 'bubanai-ng';

...
const expectedColor = { r: 200, g: 200, b: 200 };
await colorPickerDriver.setColor(expectedColor);
await waitForObjectsToBeEqual(() => 
getBackgroundColor(context, element), expectedColor);
...

With this piece of code, we ensure that we wait for the condition to be met with the minimum possible time interval (default retry interval is 0.5s between function calls).

Example 2

You have a panel from which you add elements to the builder using drag & drop. When you drag an element from the panel and press Enter during the drag, the panel should close. But if you drag an element inside the panel area, it should not close.You need to position the element beneath the panel area (i.e., perform an intermediate drag step and action).

import { getCenter, dragElementToPoint } from 'bubanai-ng';

...
const panelCenter = getCenter(panel);
const stageCenter = getCenter(stageBounding);
await dragElementToPoint(context, element, panelCenter, { tempSteps:
{ point: stageCenter, action: () => pressEnter(), }
});
...

Example 3

You need to write a test for using a hotkey. You have a lasso effect that changes the drag & drop logic when a specific keyboard combination is pressed.

import { getBoundingBox, dragElementBy, holdAndExecute } from 'bubanai-ng';

...
const elementBounding = await getBoundingBox(element);
const action = async () => dragElementBy(context, element, 100, 10);
await holdAndExecute(['Shift', MetaKeys.ControlOrCommand], action);
const boundingAfter = await getBoundingBox(element);

expect(boundingAfter.x).toBe(elementBounding.x + 100);
expect(boundingAfter.y).toBe(elementBounding.y);
...

Example 4

In the CI logs, TypeScript is transpiled into JavaScript, and the failure lines in the test differ from TypeScript (or the assertion is thrown somewhere deep in the lower level). You need to understand which functions were called at the high level and with which arguments in the test file.

import { log } from 'bubanai-ng';

export class ColorModalDriver {

@log
async setColor(color)
{...}

@log
async openModal()
{...}

@log
async closeModal()
{...}
}

Test file:

...
await colorModal.openModal();
await colorModal.setColor({ r: 20, g: 20, b: 20 });
await colorModal.closeModal();
...

Logs:

22/05/23 13:01:43 Calling ColorModalDriver.openModal();
22/05/23 13:01:53 Calling ColorModalDriver.setColor({ "r": "20", "g": "20", "b": "20" });
22/05/23 13:02:15 Calling ColorModalDriver.closeModal();

Example 5

You have a structure that can be either inside an iframe or just located in the general context of the page. The library allows you to manipulate contexts without unnecessary pain.

export class VideoPlayerDriver extends ElementBaseDriver
{...}

Test file:

...
const playerFrame = await getFrameByUrl(page, 'video-player');
let videoPlayer = new VideoPlayerDriver(page, playerFrame);// play is executed in frame context
await videoPlayer.play();
await goToPreview();

videoPlayer = new VideoPlayerDriver(page);// play is executed on page context
await videoPlayer.play();
...

Installation

To start using Bubanai in your project, simply add the dependency to your package.json:

"bubanai-ng": "^2.1.1"

To use the library, just import the necessary methods:

import { click, getText } from 'bubanai-ng';

Most methods work with Xpath, CSS selectors or ElementHandle objects.

const { type, getText, getFrameByName } = require('bubanai-ng');
const puppeteer = require('puppeteer');
const assert = require('assert');

(async () =>
{const browser = await puppeteer.launch();
const page = await browser.newPage();

const newTextValue = 'Additional content. ';
const areaSelector = 'body';
const frameSelector = 'mce_0_ifr';

await page.goto('http://the-internet.herokuapp.com/tinymce');
const frame = await getFrameByName(page, frameSelector);
const currentText = await getText(frame, areaSelector);
await type(newTextValue, frame, areaSelector, {}, { clearInput: false } ,page,);

const newText = await getText(frame, areaSelector);
assert.equal(newText, newTextValue + currentText);
await browser.close();
})();

Bubanai also provides advanced capabilities, allowing you to customize element search and perform more complex actions in Puppeteer-based frameworks. You can find more examples in the library's tests.

Documentation for the library can be found on GitHub.

Note: currently Puppeteer versions higher than 14.3.0 aren't supported.

Bubanai can be useful both for beginners and experienced automation engineers working with Puppeteer. By using this library, you can simplify your test writing process and ensure build stability.

We are actively developing the library and open to collaboration. If you have ideas, suggestions, or additions, please feel free to contribute to our project as a contributor.