Accessibility Testing with Playwright

Web accessibility (a11y) is a critical aspect of modern software development, ensuring that applications are usable by as many people as possible, including those with disabilities. As software QA engineers, we are often tasked with ensuring that our applications meet varying a11y standards. One of the most powerful tools we can leverage for this task is @playwright/test. This article will delve into the intricacies of utilizing playwright for a11y testing, offering a comprehensive guide on how to harness its capabilities effectively.

Understanding Accessibility (a11y) Testing

Before we delve into the specifics of a11y testing using Playwright, it's essential to understand what a11y testing entails. This form of testing helps identify and rectify usability issues in your application that could hinder users with disabilities. These issues could range from text with poor color contrast, making it hard to read for users with vision impairments, to UI controls and form elements lacking labels that a screen reader can identify.

However, it's important to note that while automated a11y tests can detect some common issues, many problems can only be discovered through manual testing. Therefore, a combination of automated testing, manual assessments, and inclusive user testing is recommended.

Introduction to Playwright

Playwright is a powerful, open-source framework that provides a high-level API to control web browsers. It's designed to enable reliable end-to-end testing of web applications. Though Playwright supports multiple programming languages, we will focus on TypeScript in this article.

Playwright for Accessibility Testing

Playwright offers several features that make it a valuable tool for a11y testing. One of its most significant features is the ability to capture accessibility tree snapshots, offering you an insight into how your application is perceived by assistive technologies.

import { test, expect } from "@playwright/test";

test.beforeEach(async ({ page }) => {
  await page.goto("https://ray.run/");
});

test.describe("accessibility", () => {
  test("accessibility tree", async ({ page }) => {
    const tree = await page.accessibility.snapshot();
    expect(JSON.stringify(tree, null, 2)).toMatchSnapshot("accessibility-tree.json");
  });
});

In the above code snippet, we navigate to the desired page and capture the accessibility tree, comparing it with a previous run stored in accessibility-tree.json. This comparison allows you to identify any changes in the accessibility tree that may affect a11y.

Leveraging Axe-Core/Playwright for Automated Accessibility Testing

The @axe-core/playwright package is a valuable tool in the Playwright ecosystem for automating a11y tests. It provides support for running the axe accessibility testing engine as part of your Playwright tests. Let's take a look at how to utilize this package.

import { test, expect } from "@playwright/test";
import AxeBuilder from "@axe-core/playwright";

test.describe("homepage", () => {
  test("should not have any automatically detectable accessibility issues", async({ page }) => {
    await page.goto("https://ray.run/");
    const accessibilityScanResults = await new AxeBuilder({ page }).analyze();
    expect(accessibilityScanResults.violations).toEqual([]);
  });
});

In this example, we're testing a page for automatically detectable accessibility violations. We import the @axe-core/playwright package and use Playwright's syntax to navigate to the page under test. We then await AxeBuilder.analyze() to run the accessibility scan against the page and verify that there are no violations in the returned scan results.

Advanced Accessibility Testing Scenarios with Playwright

Playwright provides several advanced features that allow you to perform more sophisticated a11y tests. For example, the AxeBuilder class supports numerous configuration options for axe. You can specify these options using a Builder pattern.

import { test, expect } from "@playwright/test";
import AxeBuilder from "@axe-core/playwright";

test.describe("homepage", () => {
  test("should not have any automatically detectable accessibility issues", async({ page }) => {
    await page.goto("https://ray.run/");
    const axeBuilder = new AxeBuilder({ page });
    axeBuilder.include("#someElement");
    const accessibilityScanResults = await axeBuilder.analyze();
    expect(accessibilityScanResults.violations).toEqual([]);
  });
});

Here, we've used the AxeBuilder.include() function to constrain an a11y scan to only run against one specific part of a page. This allows for more targeted testing based on your application's specific needs.

Landmark Testing with Playwright

Landmark testing involves checking for the existence of specific landmarks on your webpage. These landmarks are critical because they help users with assistive technologies navigate your website more effectively. Here's an example of how you can utilize Playwright for landmark testing.

import { test, within } from "@playwright/test";

test.describe("landmarks", () => {
  test("has main nav", async ({ page }) => {
    const nav = await page.$('navigation');
    expect(nav).toBeVisible();
  });

  test("has main landmark", async ({ page }) => {
    const main = await page.$('main');
    expect(main).toBeVisible();
  });

  test("has banner landmark", async ({ page }) => {
    const banner = await page.$('banner');
    expect(banner).toBeVisible();
  });
});

In this example, we're checking for the existence of a navigation, main, and banner landmark and ensuring they're visible on the page.

Incorporating Lighthouse for Accessibility Testing

While Axe provides robust automated accessibility testing capabilities, it's important to supplement it with additional tools for a comprehensive evaluation. One such tool is Lighthouse, an open-source project by Google that offers a range of audits for assessing web page quality, including accessibility.

You can access Lighthouse from Chrome by opening Developer Tools and selecting the Lighthouse tab.

To incorporate Lighthouse into your Playwright tests, you can use the playwright-lighthouse package. Here's an example of how you can run a Lighthouse accessibility audit:

import { type Browser, chromium, test as base } from '@playwright/test';
import getPort from 'get-port';
import { playAudit } from 'playwright-lighthouse';

export const test = base.extend<{}, { browser: Browser; port: number }>({
  browser: [
    async ({ port }, use) => {
      const browser = await chromium.launch({
        args: [`--remote-debugging-port=${port}`],
      });
      await use(browser);
      await browser.close();
    },
    { scope: 'worker' },
  ],

  port: [
    // eslint-disable-next-line no-empty-pattern
    async ({}, use) => {
      // Assign a unique port for each playwright worker to support parallel tests
      const port = await getPort();

      await use(port);
    },
    { scope: 'worker' },
  ],
});

test.describe('Accessibility Testing with Lighthouse', () => {
  test('should pass the Lighthouse accessibility audit', async ({
    page,
    port,
  }) => {
    await page.goto('https://ray.run/');

    await playAudit({
      page,
      port,
      thresholds: {
        accessibility: 100,
      },
    });
  });
});

In this example, we run the Lighthouse audit and assess that the accessibility score is 100, indicating that the page passes the audit.

Manual Validation and Exploratory Testing

While automated accessibility testing is a valuable tool, it's important to remember that it can only cover a portion of the WCAG (Web Content Accessibility Guidelines) requirements. To ensure a truly inclusive user experience, manual validation and exploratory testing are essential.

Manual validation involves manually inspecting and interacting with a website or application using assistive technologies, such as screen readers or keyboard navigation. This allows testers to identify any accessibility issues that may not be captured by automated tests.

Exploratory testing, on the other hand, involves exploring a website or application from the perspective of different user personas. By simulating real-world scenarios, testers can uncover potential accessibility barriers and usability issues that may impact users with disabilities.

Conclusion

Playwright is a powerful tool for a11y testing, providing a high-level API for automated and manual testing. By leveraging its capabilities, QA engineers can ensure that their applications are accessible to as many users as possible, enhancing user experience and inclusivity.