E2E testing with Playwright and Docker: Adyen sample integrations

Playwright is a modern framework that simplifies the creation and maintenance of web application end-to-end (E2E) testing. The landscape of e2e testing solutions, to be honest, is already rich with tools and frameworks, however Playwright joins the party with some distinctive features (cross-platform, cross-browser, cross-language) and a powerful maintainer (Microsoft).

This blog post presents secrets and practises of using Playwright with Docker with a specific focus on:

• use the official Playwright Docker image (with some extra tips)
• build and run a custom Playwright Docker image
• how to integrate the Playwright Docker image in the CI/CD pipeline

The article shares some of the lesson learnt by implementing E2E testing of the Adyen Sample integrations with Playwright, Docker and GithubActions

Playwright test case

First let’s look at a simple test to understand how it is written and executed.

const { test, expect } = require('@playwright/test');  

test('Card', async ({ page, baseURL }) => {     
   // check page title
   await page.goto('/');     
   await expect(page).toHaveTitle(/Checkout Demo/);  
   // click on button  
   await page.locator('text="Card"').click();    

The test can be written in various languages and run from the command line with npx playwright test.

Using the Playwright Docker image

Playwright makes available an image on Docker Hub that includes Node, all necessary dependencies, and the browsers. It is extremely convenient to run the tests in a preconfigured environment without the need for extra setup.

Run the Playwright image at the root of your project. The assumption is that the test code is found in the tests directory (otherwise change the scripts accordingly):

# run in the root of your Playwright project
docker run -v $PWD:/tests -w /tests --rm -it mcr.microsoft.com/playwright:v1.16.2-focal /bin/bash

Note the image tag focal for Ubuntu 20.04 (Focal Fossa): although there are other options (ie Ubuntu 18.04 Bionic Beaver) this works fine on Mac and Github Actions.

The above command starts an interactive bash session inside the container hosting the Playwright environment: execute the commands to add the dependencies of the application, download the browsers and kick off the test execution.

• npm install: install all dependencies of your application in the local node_modules folder.
• npx playwright install: install the supported browsers.
• npx playwright test: run all the tests.

# you are now inside the container (bash)
root@f4cfe077964d:/e2e# npm install

root@f4cfe077964d:/e2e# npx playwright install

Downloading Playwright build of chromium v939194 - 138.4 Mb [====================] 100% 0.0s
Playwright build of chromium v939194 downloaded to /ms-playwright/chromium-939194
Downloading Playwright build of firefox v1304 - 73.6 Mb [====================] 100% 0.0s
Playwright build of firefox v1304 downloaded to /ms-playwright/firefox-1304
Downloading Playwright build of webkit v1578 - 58.7 Mb [====================] 100% 0.0s
Playwright build of webkit v1578 downloaded to /ms-playwright/webkit-1578

root@a818ca0fe10a:/e2e# npx playwright test

Some tips

Let’s look at some useful customisations.

Configure the target URL

The URL of the application to be tested is defined in the playwright.config.js. It is never a good idea to hardcode it but instead aim at keeping the flexibility to test against different environments (i.e. localhost or a staging server), for example reading an environment variable:

use: {
  // bad idea
  // baseURL: 'http://localhost:8080',
  // good idea
  baseURL: process.env.URL || 'http://localhost:8080',

It is now straightforward to define the URL value at runtime:

root@a818ca0fe10a:/e2e# URL=http://myapp.com npx playwright test

Install selected extensions

There is no need to install all extensions if the tests must run only on selected browsers:

root@a818ca0fe10a:/e2e# npx playwright install chromium

Run selected tests

Specify a single test (or a subset):

# test named dropin or starting with dropin
root@a818ca0fe10a:/e2e# npx playwright test dropin

Test against localhost

Most likely the application under development (in your IDE) is running on localhost, however this makes it tricky when you want to run the Playwright Docker container to test it.

root@a818ca0fe10a:/e2e# curl -I http://localhost:8080
curl: (7) Failed to connect to localhost port 8080: Connection refused

Whaaatta!? localhost is not reachable, the reason being the Docker Networking: the localhost from inside the container doesn’t resolve to the host (unless you run on Linux with the --network=host parameter).
The first workaround is to use the built-in host.docker.internal address which is always available.

root@a818ca0fe10a:/e2e# curl -I http://host.docker.internal:8080
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 5345
Server: Werkzeug/2.0.2 Python/3.8.2
Date: Tue, 22 Mar 2022 16:24:46 GMT

Another (slightly more laborious) workaround is to map the IP address of the host (your machine) with a newly defined hostname (for example let’s call it docker).

# get IP
ipconfig getifaddr en0
xxx.xxx.xxx.xxx
# docker run adding host `docker`
docker run -v $PWD:/tests --add-host=docker:xxx.xxx.xxx.xxx -w /tests --rm -it mcr.microsoft.com/playwright:v1.16.2-focal /bin/bash
# host `docker` resolves to host
root@a818ca0fe10a:/e2e# curl -I http://docker:8080
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 5345
Server: Werkzeug/2.0.2 Python/3.8.2
Date: Tue, 22 Mar 2022 16:24:46 GMT

Important note in case of developing an Adyen integration: any chosen URL must be specified in the Allowed-Origin list to ensure the client-side requests are authorised

Create your own custom Playwright Docker image

Another option is to create a custom Playwright Docker image to “bake” configuration, dependencies and tests all together.

FROM mcr.microsoft.com/playwright:v1.16.2-focal

# copy project (including tests)
COPY . /e2e

WORKDIR /e2e

# Install dependencies
RUN npm install
# Install browsers
RUN npx playwright install

# Run playwright test
CMD [ "npx", "playwright", "test", "--reporter=list" ]

The testing suite can be executed running the Docker image and passing any env variables (i.e. URL of the application).

docker build -t custom-playwright .
docker run -it --rm -e URL=http://myapp.com --name customplaywright custom-playwright
Running 5 tests using 2 workers
✓  [chromium] › card.spec.js:5:1 › Card (3s)
  ✓  [chromium] › card2.spec.js:5:1 › Card2 (3s)
  ✓  [chromium] › dropin.spec.js:5:1 › Dropin SEPA (2s)
  ✓  [chromium] › ideal.spec.js:4:1 › iDEAL (2s)
  ✓  [chromium] › webhook.spec.js:5:1 › Webhook Notification (198ms)

Playwright E2E in the CI/CD

Building a custom Docker image makes little sense if you only run the tests locally however, if you want to include the E2E testing in the CI/CD pipeline then it gets exciting. Meet Adyen Sample integrations!

Payment Checkout example

A Sample integration is a working example of a payment checkout with the Adyen platform. It is a simple and effective open-source application that shows how the payment methods appear and execute.

Given the variety of languages and frameworks supported by the Adyen platform, we maintain more than 12 samples that are similar in terms of UI and features, but with a different backend. The containerization of the Playwright tests is therefore really valuable to us since it allows to define a single testing suite to validate multiple applications (on every git push and/or pull request).

We have effectively created on-the-fly an environment that runs both the tester application and the application being tested. At the end of the automated testing the environment no longer exists.

Let’s look at the GithubActions workflow file of one of these applications:

• first checkout the source code, build the Docker image and run it on 8080
• pull and run the Testing Suite Docker image targeting the application listening on port 8080

name: E2E (Playwright)

on:
  push:
  pull_request:
    branches: [ main ]

jobs:
  checkout:
    runs-on: ubuntu-latest
    steps:
      # checkout application code
      - name: Checkout project
        uses: actions/checkout@v2
      # build application Docker image
      - name: **Step 1: build image (application)**
        run: docker build -t test-image:latest .
      # run application Docker image
      - name: **Step 2: start container (application)**
        run: docker run --rm -d --name test-image -p 8080:8080 -e ADYEN_API_KEY="${{ secrets.ADYEN_API_KEY }}" -test-image:latest
      - name: **Step 3: run Testing Suite container**
        # run testing suite Docker image
        run: docker run --rm --name adyen-testing-suite --network host ghcr.io/adyen-examples/adyen-testing-suite:latest

Conclusions

I hope the article helps to shed some light on Playwright with Docker, which is not extensively (yet) used or documented. The valuable takeaway here is, using Docker, it is possible to design custom E2E workflows portable between development environments and CI/CD engines.

If you enjoyed the article follow me and Adyen Developers for more articles and tutorials. Happy testing!

References

Adyen integration samples on Github
Adyen Testing suite on Github
Playwright on Docker page