Fix Your Flaky End-To-End Tests With Testing Library

End-to-end tests for web applications tend to get a bad reputation for failing consistently. A quick online search for "end-to-end tests flaky" yields tons of blog posts about how fragile these kinds of tests are. There are even plenty of posts of people and organizations who give up on end-to-end tests altogether.

This reputation isn't wholly unearned, however. End-to-end tests can be a pain to deal with during development. It comes with the territory, given the ground these tests cover. When lots of interactions and moving parts come into play, a single point of failure can bring everything crumbling down with a big, fat FAILED message.

Still, it gets incredibly frustrating when your tests fail when the functionality under test is the same. There are plenty of reasons why a full end-to-end test can fail for reasons other than functionality changes. One of the main reasons - if not the main reason - for failure is due to simple UI changes.

The way most web testing frameworks do their work is by looking up specific elements on a web page with element selectors. These selectors are often reliant on the implementation of those elements in the markup that generates the page. It means you need to know the element's ID or other attributes like as a class name, so your test knows what it needs.

The problem comes when someone makes a small change to the interface that's being. If a developer changes a specific ID or attribute that the test looks for without updating the tests, it causes the test to fail since it can't find the element. Usually, these UI changes have no bearing on the functionality of the application. These failures are common and lead to wasted time and frustration.

There are also some issues in some modern web applications, where elements get dynamically generated. Since the testers won't know ahead of time how to find a specific element on the page, it becomes messy writing selectors to find one of these dynamic elements. These selectors are also very fragile since they often rely on the page's structure, making it easier to break tests.

Find your elements using Testing Library

To minimize testing issues caused by changes in an application's implementation, a set of utilities called Testing Library can help.

Testing Library is a collection of utilities providing methods that help select elements on a given page in a better way than using ID or classes. Instead of finding elements by a specific selector, you can use more-readable methods like finding input fields by label or selecting a button by its text. These methods minimize the risk of UI changes breaking your tests because it looks up elements in a more "human" way.

Note that it minimizes the risk, not eliminate it. The risk of UI changes breaking your tests is still present with Testing Library. However, with Testing Library, there's a higher possibility that a UI change breaking a test means that something functionally changed.

An example of a potential change in functionality after a UI change is when the text of a button changes. Usually, the text for a button indicates what it does. If that text for the button changes, it might signify a change in functionality. It's an early alert to figure out if the functionality under test needs to change.

Despite its name, Testing Library is not a single library, but more of a family of libraries. Its core library is called the DOM Testing Library, which contains the main methods of querying and interacting with a web page. This library is the basis for using Testing Library in lots of different JavaScript frameworks. There are libraries for React, Vue, Angular, Cypress, and much more.

Using Testing Library with TestCafe

This article covers the basics for getting started Testing Library using TestCafe as our test framework of choice.

A few weeks ago, Dev Tester covered how to get started with TestCafe. The article serves as an introduction to the framework, containing a few examples covering essential usage as a starting point. We'll use those tests to demonstrate how to use Testing Library in TestCafe. You can read the article to learn how to create the tests from scratch, or you can find the finalized code for that article on GitHub.

To begin using Testing Library for our TestCafe tests, we'll need to install and set up the TestCafe Testing Library package. This package allows you to use the Testing Library methods inside TestCafe.

To install the package, all you need to do is to run the command npm install @testing-library/testcafe inside the directory where the tests are.

After installing the package, you need to set up the library. Testing Library needs to inject some code on the pages under test for its methods to work correctly across different browsers and test environments. To tell TestCafe to inject what Testing Library needs, we need to set up a configuration file.

When running TestCafe tests, the test runner first checks for the presence of the .testcaferc.json file in the project's root directory. TestCafe applies any configuration settings here to your tests.

In this example, we need to use the clientScripts setting to inject the Testing Library scripts for all your tests. Create a new file called .testcaferc.json in the root directory for your tests and save the following:

{
  "clientScripts": [
    "./node_modules/@testing-library/dom/dist/@testing-library/dom.umd.js"
  ]
}

This configuration setting looks for the necessary scripts from the Testing Library package that we installed and injects them automatically when we run our tests.

With this set up completed, we're ready to use Testing Library. Our TestCafe tests now have the Testing Library API available for use.

Looking up elements with Testing Library

Let's check out how Testing Library works by updating our tests. First, let's use the simple test we have for verifying the Airport Gap home page. This test opens the Airport Gap home page and verifies that it contains an element with specific text.

The test only has one selector, defined in its page model (page_models/home_page_model.js):

import { Selector } from "testcafe";

class HomePageModel {
  constructor() {
    this.subtitleHeader = Selector("h1").withText(
      "An API to fetch and save information about your favorite airports"
    );
  }
}

export default new HomePageModel();

Let's change that selector to use Testing Library instead:

import { getByText } from "@testing-library/testcafe";

class HomePageModel {
  constructor() {
    this.subtitleHeader = getByText(
      "An API to fetch and save information about your favorite airports"
    );
  }
}

export default new HomePageModel();

We made two changes to this page model class. The first change done is importing the getByText method from TestCafe Testing Library. This method searches for a node on the web page that contains the text content specified when calling the method. We won't use the Selector method anymore so we can remove that import statement.

The other change was to the subtitleHeader property. Here, we'll use the getByText method to find the subtitle using its text. Note that we don't need to search for a specific element as we did before, looking for an h1 element. Testing Library doesn't care what type of element it is, just what it does. In this case, we want to find something that has specific content.

If you re-run the home page test (npx testcafe chrome home_test.js), the test passes. Functionally, this test works the same as before. However, the changes are a bit of an improvement. If someone decided to change the element from an h1 to an h2 element, the test would break even though the text is still there.

In all fairness, there's still a possibility of tests breaking because of a text change. However, this test is a very simple example and isn't a particularly useful example of a real-world test. Your end-to-end tests should not merely look for some basic text. Still, it's an excellent example to demonstrate how easily an end-to-end test can break and how Testing Library helps minimize these issues.

Filling out forms with Testing Library

Let's do something a little more with Testing Library to demonstrate its usefulness better. The other test we have validates the login functionality of Airport Gap. It loads the login page, fills out and submits the form, then verifies that we logged in successfully.

The page model for this test (page_models/login_page_model.js) contains four selectors:

import { Selector } from "testcafe";

class LoginPageModel {
  constructor() {
    this.emailInput = Selector("#user_email");
    this.passwordInput = Selector("#user_password");
    this.submitButton = Selector("input[type='submit']");
    this.accountHeader = Selector("h1").withText("Your Account Information");
  }
}

export default new LoginPageModel();

Using Testing Library, let's update the selectors and see how it looks:

import { getByLabelText, getByText } from "@testing-library/testcafe";

class LoginPageModel {
  constructor() {
    this.emailInput = getByLabelText("Email Address");
    this.passwordInput = getByLabelText("Password");
    this.submitButton = getByText("Log In");
    this.accountHeader = getByText("Your Account Information");
  }
}

export default new LoginPageModel();

Here we have more interesting changes. We're using the same getByText method we used in the previous test for finding the submit button and account header text. However, we are adding a new method: getByLabelText. This method works by finding the label with the given name and then looks up the element associated with that label.

Once again, if you run the test, the test passes.

Why look up form elements by label text?

If you check out the Testing Library API, there are other ways to search for input elements, such as getByPlaceholderText. However, the recommended way to search for input elements by its label, if possible.

Searching for elements by the label has the additional benefit of ensuring that your labels are appropriately associated with form inputs. Having explicit or implicit label associations is essential for accessibility, helping remove barriers for people with disabilities.

For more information on which query is most appropriate for your use case, read the Which query should I use? page in the Testing Library documentation.

Tips for minimizing risk with Testing Library

In all of the examples above, there's still the potential of UI changes breaking a test. For example, if someone changed the label "Email Address" label for the login form to something like "Company Email," the test would fail since it couldn't find the selector.

There are a few tips you can employ to your tests and application to further minimize the risk of implementation changes breaking your tests:

• Use regular expressions instead of looking for exact text. When using getByText with a string, it searches for the exact text by default. However, you can use a regular expression to find a substring instead. For instance, instead of "Email Address", you can use /email/i to search for an element containing "email" anywhere in its content. Be aware that if you have multiple elements with the same term, your regular expression may not find the element you want.
• Use specific attributes that are less likely to change. Some Testing Library methods, like getByLabelText, can search for different attributes. For example, getByLabelText searches for the specified string in the for attribute, the aria-labelledby attribute, or the aria-label attribute. These attributes are less likely to change compared to searching for the label content itself.
• Use the getByTestId method. This method looks for elements containing the data attribute data-testid. This data attribute only serves as an identifier for your tests and won't affect how the element shows up on your page. Since its only use is for looking up elements for testing purposes, the attribute can contain any value and shouldn't need any changes even if the element changes drastically. It's also ideal for pages with dynamic content. The only downside is that you need access to the application's code to set up these attributes in the application itself.

Summary

End-to-end tests tend to be a bit more fragile than other kinds of testing. It's the nature of the beast, given how much coverage these tests provide. However, you can take some steps to reduce failures in your tests.

The methods provided by the Testing Library API help prevent unnecessary test breakage due to implementation changes that don't change your application's functionality. With Testing Library, you can look up elements in a way that's closer to how people look for them on a page. You don't need to worry about IDs, class names, or figuring out how to select a dynamic element.

The examples in this article describe the basics for Testing Library. The changes made to the tests we started with are minimal but cover most of how the library makes your end-to-end tests less prone to failure. In larger projects, the benefits are more apparent. Testing Library saves you and your team lots of wasted time and frustration.

What other problems caused your end-to-end tests to break frequently? How have you dealt with these issues? Let me know in the comments below!

The source code for the examples in this article is available on GitHub.

P.S. Was this article helpful to you? I'm currently writing a book that will cover much more about about the TestCafe testing framework.

With the End-to-End Testing with TestCafe book, you will learn how to use TestCafe to write robust end-to-end tests on a real web app and improve the quality of your code, boost your confidence in your work, and deliver faster with less bugs.

For more information, go to https://testingwithtestcafe.com. Sign up and stay up to date!