How to run Playwright tests with Selenium Grid versions after `4.5`

Compared to other automation testing libraries, Playwright is relatively new, but it supports writing tests in different languages.

If you're using Java, Playwright can take advantage of the Selenium Grid to scale test execution, helping you save time and increase efficiency.

TL;DR;

• Playwright has a limitation that restricts running tests in Selenium Grid to only the version prior to 4.5.2-20221021 (at the time of this writing - February/2023).
• Selenium 4 has passed through a redesign of its architecture. After a long period of stabilization, a long-standing intermittent issue was fixed in version 4.5.2-20221021.
• If Playwright tests only work with versions prior to 4.5.2-20221021, it won't be able to benefit from the latest improvements made by the Selenium team to the grid or any browser version upgrades. A workaround for the issue is to use a specific property -Djdk.httpclient.websocket.intermediateBufferSize, passed as an argument to the selenium-server.jar.

Problem

Although Playwright is still relatively new compared to Selenium, it generally offers faster test execution times. In a "real-work" test that I used to compare the two tools, the results were as follows:

The example above is a simple test run from a local machine, without using Selenium Grid.

According to the Playwright website, tests written with Playwright can run against Selenium Grid, as documented here. However, as of February 2023, there is currently an issue with Selenium Grid versions 4.5.2-20221021 or later, which can cause errors when attempting to use Playwright with Selenium Grid.

• Cannot run playwright tests on Selenium Grid

If Playwright tests can run only with Selenium Grid version prior to this, then it means it is running only with older versions of Google Chrome or Microsoft Edge:

History

When Selenium 4 was release, it changed completely how Selenium Grid is structured. It has totally changed its design, based on distributed components to handle messages between the hub and the nodes.

After its release, a critical intermittent issue received significant attention, as it was difficult to reproduce and persisted for almost two years before being resolved.

• https://github.com/SeleniumHQ/selenium/issues/9528#issuecomment-947994758

This fix was officially released on docker-selenium project on version 4.5.2-20221021. The exact same version that started to present issues with the execution of tests written with Playwright.

Fix (Workaround)

The fix was reported by EndPositive in an issue opened for the Selenium team.

The issue seems to be related to WebSocket communication, with messages larger than the default configuration of Selenium Grid causing problems.

The suggestion is to increase the WebSocket buffer size passing the parameter Djdk.httpclient.websocket.intermediateBufferSize to the selenium-server.jar.

Using docker-selenium images, it is possible to do as shown in the example below:

version: "3"

services:
  selenium-hub:
    image: selenium/hub:4.8.0-20230210
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"
    environment:
      - JAVA_OPTS=-Djdk.httpclient.websocket.intermediateBufferSize=3000000
      - SE_EVENT_BUS_HOST=selenium-hub

  chrome:
    image: selenium/node-chrome:4.8.0-20230210
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - JAVA_OPTS=-Djdk.httpclient.websocket.intermediateBufferSize=3000000
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - HUB_HOST=selenium-hub
      - HUB_PORT=4444
      - SE_NODE_GRID_URL=http://localhost:4444

To run Playwright tests against Selenium Grid, the environment variable SELENIUM_REMOTE_URL must be declared:

export SELENIUM_REMOTE_URL=http://localhost:4444

After that, tests can be run in different ways:

./gradlew test

Or

./mvnw test

Or with other build tools of preference.

Final consideration

It seems that Playwright uses Selenium Grid, but Selenium Grid does NOT officially support Playwright. The problem to run Playwright tests with the newest version of Selenium Grid were raised in Selenium GitHub project, however it hasn't been investigated so far to be fixed once it cannot be reproduced with tests implemented using Selenium (and Selenium WebDriver).