#003 | Let's Automate Playwright Testing With GitHub Actions

This post was originally published on the Learn Playwright blog. Reference the original post to get the most updated version of this content.

🔖 | Today's Resources

• Recipes4AJ Project Repo - explore demo app source!
• End to End Testing With Playwright - Playwright + Azure Static Web Apps
• Playwright Test - Command line tool and options
• Continuous Integration - options and examples for Playwright Test usage.
• Demo.Playwright - repo for demo of Playwright capabilities + CI pipeline.

🗺 | Article Roadmap

Here are the steps covered in this post:

• 1️⃣ | Building and deploying a simple demo app
• 2️⃣ | Why Integrate Playwright?
• 3️⃣ | Let's Setup Playwright Tests
• 4️⃣ | Customize Tests
• 5️⃣ | Run Tests Locally
• 6️⃣ | Automate Testing with GitHub Actions
• 7️⃣ | Continuous Integration Options

Plus, review resources and get a sneak peek at next steps.

🎯 | Today's Objectives

I wanted to explore Playwright Test tools in more detail. But it occurred to me that doing this with a somewhat realistic app, as an experiment sandbox, might be valuable. So my day 3 goals became:

• Build and deploy a simple demo app
• Instrument that app to use Playwright for end-to-end tests
• Automate testing workflow using GitHub Actions

Once achieved, I'd have the basic infrastructure in place to start exploring Playwright authoring, debugging and profiling tools in a concrete way!

1️⃣ | Building and deploying a simple demo app

This immediately brought up a few questions - what app should I build? What front-end technology should I use to build it? Where should I host it - and how could I quickly setup a build-deploy workflow so I could start exploring test integrations later?

For the last step, I was inspired by this amazing 20-minute talk at the recent Static Web Apps: Code To Scale virtual event.

So I decided to go with Azure Static Web Apps for my fun demo project. And since I'm a huge fan of Hugo for creating static sites (especially blogs), I decided to use the Hugo Quickstart tutorial to make this happen.

My use case - inspired by a pandemic-driven cooking obsession - was to convert my tweet-sized recipes into a blog with recipe cards. A morning's worth of fun exploration - and I had that done! Say hello to Recipes For AJ. You can learn more about how I built that in this post:

Let's Build A Recipe Blog with Azure Static Web Apps & Hugo

Nitya Narasimhan, Ph.D for Microsoft Azure ・ Dec 7 '21

#staticwebapps #gohugo #beginners #githubactions

2️⃣ | Why Integrate Playwright?

One goal was to use the app as an experimental sandbox to explore Playwright test authoring, debugging and profiling tools. Get a sense for how these worked using a real-world application and test environment.

A second goal was to start thinking about what tests we could do on an app like this - and then figure out how to author those tests in Playwright. Some thoughts on what we could explore here:

• Cross-Browser - verify app works on Chromium, WebKit and Firefox
• Mobile - use emulation to check responsiveness of app design
• Content - make sure all posts contain specific section or tag
• Navigation - check that all links in post are valid
• Accessibility - check if site meets web accessibility guidelines
• Performance - trace views to detect and fix inefficiencies

But let's start by first making sure we have Playwright setup for the project.

3️⃣ | Let's Setup Playwright Tests

I started with something simple and used the Scaffold e2e tests for project approach. And for convenience (see Troubleshooting later) I chose to setup Playwright in a testing subfolder within the app project.

Here is what that looked like (output truncated for clarity):

$ npm init playwright testing

Getting started with writing end-to-end tests with Playwright:
Initializing project in 'testing'
✔ Do you want to use TypeScript or JavaScript? · JavaScript
✔ Where to put your end-to-end tests? · e2e
✔ Add a GitHub Actions workflow? (Y/n) · false
Initializing NPM project (npm init -y)…
Wrote to <..>/testing/package.json:

{
  "name": "testing",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Installing Playwright Test (npm install --save-dev @playwright/test)…
...
...
...

We suggest that you begin by typing:

  cd testing
  npx playwright test

And check out the following files:
  - ./testing/e2e/example.spec.js - Example end-to-end test
  - ./testing/playwright.config.js - Playwright Test configuration

Visit https://playwright.dev/docs/intro for more information. ✨

Happy hacking! 🎭

Visit the GitHub repo to see the result - and note these two items:

• an e2e directory containing the default test script.
• a playwright.config.js file with default test projects.

TROUBLESHOOTING

Here are a couple of decisions I made, that helped get around some issues I faced during the setup process:

1. Setup Playwright in its own subfolder (here testing/) for simplicity. Otherwise the presence of Node.js config file (like package.json) causes an unexpected conflict with the Azure Static Web Apps workflow, causing that Action to fail.

2. Playwright setup gives you the option to have a GitHub Actions file created. I declined it so I could (manually) update the one I already had for this project. If you choose to do otherwise, make sure you add a dependency on that build/deploy job so this runs only after that succeeds.

4️⃣ | Customize Tests

The default playwright configuration is in testing/playwright.config.js - make changes here if you want to configure projects for cross-browser testing (e.g., add mobile targets) or configure other test options.

I elected to make only one change - turning trace recording on by default, for every run (rather than only on first retry). This ensures that a trace.zip file is created on each run for analysis.

    /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
    trace: 'on' // 'on-first-retry',

The default end-to-end tests are in the testing/e2e/example.spec.js file - and do a basic test that navigates to the Playwright site, checksif the title matches expectations, and clicks an element (matching a text selector).

This is a good time to just modify these to match your own deployed Azure Static Web Apps site. Here's what mine looks like:

// @ts-check
const { test, expect } = require('@playwright/test');

test('basic test', async ({ page }) => {
  await page.goto('https://bit.ly/recipes-for-aj');
  await expect(page).toHaveTitle("Recipes 4 AJ");

  await page.locator('text=Tags').click();
});

It's a trivial change, but it starts the process of connecting the test suites to the current project. We'll make more changes to these files later as we explore Playwright API and tooling.

5️⃣ | Run Tests Locally

With this setup, you should be able to run the end-to-end tests in your local environment immediately. Try this out:

$npx playwright test
Using config at <..>/aswa-hugo-recipes4aj/testing/playwright.config.js

Running 3 tests using 3 workers

  3 passed (4s)

To open last HTML report run:

  npx playwright show-report

The default configuration is setup to create an HTML report of the test run - and since we turned on traces, it should also have a link to the trace report that we can analyze in the TraceViewer.

For now - let's just run that last command and see what happens:

$ npx playwright show-report

  Serving HTML report at http://127.0.0.1:9323. Press Ctrl+C to quit.

In the next blog post, we'll dive into more details on what this report shows - and explore other local testing tools in more detail.

6️⃣ | Automate Testing with GitHub Actions

Now, we just need to make sure these tests are run automatically as part of our build-deploy workflow. The Azure Static Apps setup provides a default GitHub. Actions file with a Build and Deploy job defined. We just need to add a Test site using Playwright job that is run after the build/deploy completes successfully.

This is what my updated GitHub Actions file project looks like - I've truncated sections for clarity so we can focus on the test job definition.

name: Azure Static Web Apps CI/CD

on:
  push:
    branches: 
      - main
  pull_request:
    types: [opened, synchronize, reopened, closed]
    branches:
      - main

jobs:

  build_and_deploy_job:
  // TRUNCATED FOR CLARITY

  test:
    name: "Test site using Playwright"
    timeout-minutes: 60
    needs: build_and_deploy_job
    runs-on: ubuntu-20.04  
    steps:
      - uses: actions/checkout@master
      - uses: actions/setup-node@v2
        with:
          node-version: '14.x'

      - name: Install dependencies
        run: | 
          cd testing
          npm ci
          npx playwright install-deps
          npx playwright install
      - name: Run Playwright Tests        
        continue-on-error: false
        working-directory: testing       
        run: |   
          npx playwright test e2e --reporter=html --config playwright.config.js


  close_pull_request_job:
  // TRUNCATED FOR CLARITY

Note three things:

• The timeout-minutes defines the longest a job will run before GitHub cancels it. The Playwright docs example suggested 60 minutes.
• The needs parameter identifies jobs that must complete before this can run. Our dependency is on the build_and_deploy_job.
• The working-directory parameter should be set to testing to ensure test runs set the right working context.

Now save the updated workflow file and commit it to the repo.

• You should immediately see the Actions run! Note how the visual reinforces the dependency between the Build and Deploy Job and Test site using Playwright jobs in execution order.

• Click on the job record to see a break down of steps - each showing the actions taken, and the time taken to complete it.

7️⃣ | Continuous Integration Options

Want to learn more and explore alternative CI/CD services?

The Playwright documentation has a section on Continuous Integration with advice for configuring various systems including Azure Pipelines and GitHub Actions.

Also check out this excellent Demo.Playwright example using the playwright container environment and running richer actions (e.g., upload and publish reports) as part of the pipeline.

🌟 Day 3 :Review

🔥 Congratulations! We now have a Hugo-themed blog hosted on Azure Static Web Apps, with Playwright End-to-End testing workflow setup. Plus, build-deploy-test GitHub Actions workflows to streamline our development!

That was a lot to digest - let's recap!

• [X] Learned about Azure Static Web Apps (ASWA)
• [X] Watched E2E Testing With Playwright and ASWA
• [X] Built & Deployed a Hugo-themed Static Web Site
• [X] Instrumented it to use Playwright E2E Testing
• [X] Explored Continuous Integration options for Playwright
• [X] Setup Playwright Tests & Actions to automate build-deploy-test.
• [X] Maybe even learned a new recipe!

🌟 Day 4 : What's Next?

Let's revisit our previous objectives to explore the tools available for authoring, debugging and analyzing Playwright Test runs. In the upcoming days, we will explore the following:

• Trace Viewer - dive into the trace report generated
• Reporters - understand show-report usage and customizations.
• Test Generator - how can we generate tests out of the box?

And just to inspire you to do the same, I'll leave you with this tweet!

This post was originally published on the Learn Playwright blog. Reference the original post to get the most updated version of this content.