Testing for Accessibility with Playwright

Merging my favorite tool with my passion: 🎭 + A11y

For my first accessibility tests, I decided to start with a couple of ready-made scripts provided in the documentation. These tests use the @axe-core/playwright package to evaluate a wide range of accessibility rules.

Some of these rules align with specific criteria from the Web Content Accessibility Guidelines (WCAG), while others are considered best practices but not explicitly required by WCAG.

Testing for WCAG A Compliance

If you are interested in scanning an entire page for accessibility violations, the following test is a good start:

test.describe('homepage', () => { 
  test('should not have any automatically detectable accessibility issues', async ({ page }) => {
    await page.goto('https://corinamurg.netlify.app/'); 

    const accessibilityScanResults = await new AxeBuilder({ page }).analyze(); 

    expect(accessibilityScanResults.violations).toEqual([]); 
  });
});

If you prefer to focus on rules that correspond to particular WCAG criteria, you could use AxeBuilder.withTags().

I also started with AxeBuilder.withTags(). My targets were the WCAG A guidelines for which I used the wcag2a and wcag21a tags.

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test('should not have any automatically detectable WCAG A violations', async ({ page }, testInfo) => {
    await page.goto('https://corinamurg.netlify.app/');

    const accessibilityScanResults = await new AxeBuilder({ page })
        .withTags(['wcag2a', 'wcag21a'])
        .analyze();

    expect(accessibilityScanResults.violations).toEqual([]);
});

I did not expect to see any violations, and indeed there were none. Moving on!

Unexpected Results with WCAG AA

A different story with WCAG AA! Again, I did not expect any violations, but there were quite a few!

test('should not have any automatically detectable WCAG AA violations', async ({ page }, testInfo) => {
    await page.goto('https://corinamurg.netlify.app/');

    const accessibilityScanResults = await new AxeBuilder({ page })
        .withTags(['wcag2aa', 'wcag21aa'])
        .analyze();

    expect(accessibilityScanResults.violations).toEqual([]);
});

I had previously run the axe Dev Tools scan and the page received a clean bill of health:

But the Playwright scan revealed SIX(!) elements with color contrast issues:

• three with the sample-project--title class, and
• three with the sample-project--description class.

All failure summaries were similar to this one:

+ "failureSummary": "Fix any of the following:
+    Element has insufficient color contrast of 2.61 
       (foreground color: #a0a0a0, 
       background color: #ffffff, 
       font size: 14.4pt (19.2px), 
       font weight: normal). 
       Expected contrast ratio of 4.5:1",
 +   "html": "<p class=\"sample-project--description\">
                    React, Typescript
              </p>",
 +    "impact": "serious",
 +    "none": Array [],
 +    "target": Array [
 +    ".my-portfolio-site > .sample-project > .sample-project--description",
 +         ],

Troubleshooting the Color Contrast Issue

It was quite the head-scratching moment because the CSS declaration blocks told a different story.

.sample-project--title {
    align-self: start;
    display: inline-block;
    font-size: 1.7rem;
    font-weight: 500;
    margin-top: 32px;
    padding-bottom: 10px;
    position: relative; 
}

.sample-project--description {
    font-size: 1.2rem;
    font-weight: 400;
    margin-top: 0;
}

To determine the color contrast of an element we need to know the foreground and the background colors. In this case, neither element has specific colors, so they inherit from the body element: white background, and black text. Doesn't this combination give the best color contrast one can ask for?! 🤔

And yet, the test revealed a color-contrast violation … Not only that, but it was referring to a text color that my site does not even use: foreground color #a0a0a0.

This actually gave me the idea to be more specific about the colors for each element. So, I added color:inherit. And … surprise! It worked ... for the sample-project--title elements, that is! Running the test again revealed only 3 errors, one for each of the three sample-project--description elements.

After this semi-victory, I assumed the test needed more specificity so I replaced color:inherit with the actual hex values. No luck! (and in hindsight, no surprise. I was at a point where I was just guessing …) 😕

Looking for Help from the WebAIM’s Contrast Checker

Was the size of the text too small? I wouldn’t have thought that to be the cause of the violation, again because of the black text on white background combination. Still, I checked WebAIM’s color contrast tool just to make sure (and because I did not know what else to do!!!):

Here’s also WebAIM’s specification about the relationship between font size and expected color contrast:

WCAG 2.0 level AA requires a contrast ratio of at least 4.5:1 for normal text and 3:1 for large text.

WCAG 2.1 requires a contrast ratio of at least 3:1 for graphics and user interface components (such as form input borders).

WCAG Level AAA requires a contrast ratio of at least 7:1 for normal text and 4.5:1 for large text.

Large text is defined as 14 point (typically 18.66px) and bold or larger, or 18 point (typically 24px) or larger.

Back to my trouble-making elements: with a font size of 19.2px and a contrast ratio of 21:1, they should pass not just the AA guidelines, but the AAA ones as well!

Will AAA Fail Too?!

Sigh of relief … running the WCAG AAA scan found no errors with those particular elements. But wait ... why not? They failed the lower standard. How come they passed the higher standard?! 😲

There were a few errors regarding another elements, and for good reasons. This is an example of one of the failure summaries:

+ "failureSummary": "Fix any of the following:
+    Element has insufficient color contrast of 4.5 
       (foreground color: #1a73c6, 
        background color: #f6f6f6,
        font size: 12.0pt (16px), 
        font weight: normal). 
        Expected contrast ratio of 7:1",
+    "html": "<span class=\"footer-heading\">Profiles</span>",
+    "impact": "serious",
+    "none": Array [],
+    "target": Array [
+           ".footer-right--social > .footer-heading",
+         ],

This is the CSS declaration block for one of the elements with the AAA violations:

.footer-heading {
    background-color : #f6f6f6
    color: #1a73c6;
    font-size: 1rem;
    font-weight: 500;
    text-transform: uppercase;
}

At 16px, the font-size for this element is considered “normal” so it needs a contrast ratio of 7:1 to meet the AAA guideline. The failure summary confirmed (correctly this time!) the contrast checker’s result: the element’s contrast ratio is only 4.5:1, good enough for AA but falling short of the AAA guidelines.

Back to the WCAG AA failed test …

How much time is too much time investigating and troubleshooting a problem when you have no idea why it exists in the first place? After the failed attempts mentioned above, I scoured the documentation, reached out to Playwright’s Discord community, and finally found the courage to open a bug issue on Playwright’s GitHub repo. It had to be a bug, there was no other explanation I could find.

This was a big step for me. I was suddenly stepping into a space where everyone was a pro, and dreaded the idea of embarrassing myself. I read (and re-read!) the contributing guidelines, then (very, very) carefully followed the instructions in the bug issue template, and completed one. After a long wait and a good self-pep talk (“it’s not the end of the world!”), I clicked submit.

Check out my masterpiece of an issue on Playwright's GitHub repo.

The response was swift:

This sounds like a bug which should be reported at @axe-core/playwright or axe-core since Playwright does not do anything related to axe.

Ahhh, the embarrassment I dreaded … If it’s indeed a bug, as I had assumed, it must be from the package that actually checks for violations. Why did I not think about that?!

Another round of electronically leafing through another documentation. So far, no answers. But will soon find a way to submit a bug report. Now that I conquered my fear of public scrutiny, there’s no stopping me! 😎

Coming Up

In the mean time, I have another a11y bug to deal with. Found by Google’s Lighthouse extension (ignored by axe Dev Tools!):

Can you guess my next challenge?

Write a test in Playwright that will check for the following WCAG guideline:

For each user interface component that includes a visible text label, the accessible name MUST match (or include) the visible text in the label.

Let’s go Playwright! Get ready JavaScript!

Resources

WebAIM Contrast Checker

Debbie O’Brien’s Playwright Series on dev.to

Playwright Documentation

Image credit: Photo by Alex Kondratiev

Description: The image shows a hand pouring a blue liquid from a glass test tube into a flask with red liquid, and another hand pouring a green liquid into the flask. The background is plain white.