Creating Effective Automated Accessibility Test Cases

One of the biggest fallacies in UI testing, especially when it comes to accessibility testing, is simply just making everything into a checkbox. Did we do the minimum UI testing on that component? Check! Did we do accessibility testing? Check!

The tests that are written may not be the most thought out or effective for actually testing the UI. This can cause UI bugs to persist because the tests do not have the proper depth or requirements.

UI accessibility testing also is at fault for this. The typical phrase heard is "Yeah we do accessibility testing! Its fully integrated and we use it each time our tests run!". However, most development teams continue to push out automated accessibility issues.

How can we improve our accessibility UI tests to more effectively know the issues in our code?

Proper Testing Structure

Instead of rushing to make a test case that says "accessibility check", there are ways to structure and build regression tests for more effective accessibility testing.

One to One UI Tests

There is a principal in UI testing that says for each piece of UI there should be a test case to follow. Meaning one piece of UI, one test case. This ensures that each component or piece of HTML has a particular set of regression tests.

We can apply this same principal to the accessibility test cases we setup. Instead of just simply installing axe-core or pa11y and running a scan at a high level, we can effectively place the scans for better coverage.

For example, if we have a React component, that has a set of unit tests for its UI functionality, we can create a test case for that component like so:

describe('Footer Component', () => {

  test("Funtionality - Component has loaded", () => {
    render(<Footer />);
    expect(screen.getByRole("heading")).toHaveTextContent("Contact Us");
  });

  test("Functionality - List number", () => {
    render(<Footer />);
    expect(screen.getByTestId("footList")).not.toEqual(null);
    expect(screen.getByTestId("footList").childElementCount).toBe(4);
  });


  test("Accessibility - base render",async () => {
   const {container} = render(<Footer />);
   const results = await axe.run(container)
   expect(results.violations.length).toBe(0);
  });

In that same test spec, if the UI test case expands and collapses the component, you could run a scan in each of those states and collect the results for each state of the UI. The goal is to make the accessibility test apart of the testing structure, but also effectively show the issues!

Multiple Levels of Test

If your development teams multiple levels of test such as Unit and Integration level testing, you should include accessibility testing at both levels.

Why? If you test only in your unit or component UI tests, you will miss more issues that exist when those component begins interacting on the page together.

An example of a component that would not have any automated issues in unit test but would in integration would be a component that has a "more info" link. If that component is used multiple times on the page, then in integration testing you would find multiple links with the same text!

Separate Accessibility Suite

Different than having your accessibility tests as part of your test cases, you can have a separate suite of UI tests for accessibility. Development teams that are newer to including accessibility in their UI tests tend to go this route simply because their suite may be too complex to add it in or its easier for them to get it setup.

Although not as effective as including in your current testing structure, there are benefits to creating a separate set of tests for accessibility only, which include:

• More streamlined reporting on the state of accessibility
• Easier to see results
• Easier to create JIRA bugs

To do this, in your project, create a separate accessibility tests folder and create either a single spec or specs broken down by business logic (groups of pages, etc). Then in your test describe give it a name similar to "Accessibility of " and then test each page in the experience.

describe("Accessibility of Space Jam",  () => {

  test("1996 Page", async () => {
    await pa11y('https://spacejam.com/1996').then((a11yResults) => {
        expect(a11yResults.issues.length).toBe(0)
    });
  });

  test("2000's Page", async () => {
    await pa11y('https://spacejam.com').then((a11yResults) => {
        expect(a11yResults.issues.length).toBe(0)
    });
  });

});

Add Accessibility Regression Tests

Simple javascript libraries can give you easy, static accessibility results. However, you can go a step further and include specific regression tests for accessibility issues.

When writing your teams test case, instead of just including a simple axe or pa11y test, build an accessibility regression test as part of your UI requirement testing.

Let's take a look at an example using Cypress and a component that expands and collapses. I can write a simple accessibility regression test that ensures that I am properly setting the aria-expanded true/false like so:

   it('A11y - FAQ proper expanded state', () => {
      cy.get('#toggle').invoke('attr', 'aria-expanded').should('eq', 'false');

      cy.get('#toggle').trigger('keydown', {
         key: 'Enter',
       })
      cy.wait(150);
      cy.get('#toggle').invoke('attr', 'aria-expanded').should('eq', 'true');

      cy.get('#toggle').trigger('keydown', {
         key: 'Enter',
       })
      cy.wait(150);
      cy.get('#toggle').invoke('attr', 'aria-expanded').should('eq', 'false');
  });

Now these test cases can be apart of the standard requirement testing. The only thing to ensure is in the test the title includes "A11y" or "Accessibility" to ensure easily findable results.

In Summary

It's time to get past the "we included accessibility tests" that get little to no results and more effectively create test cases that allow your development team to fix more accessibility issues now! Using the suggestions above your team can easily begin to maintain an accessible application for years to come.