Internal Access Properties : Encouraging less brittle end-to-end testing

Summary

When testing with web components you don't own that have shadow roots, its a much better experience for those components to expose first-class properties that expose DOM elements you'll likely need than for you to go digging in the shadow root with querySelector. Adding first-class internal access properties makes tests written using third-party components WAY less brittle and keeps the SemVer contract intact.

Testing with a shadow DOM component you don't own

Imagine you're consuming a library of web components written by someone else, say the amazing ones from Shoelace or Adobe Spectrum and you have written some pretty complex user flows that you'd like to fully test end-to-end, putting yourself in the user's (automated) shoes and simulating clicks through your UI to make sure that your flows are working as intended as your user makes decisions.

And, for the sake of argument, lets say that your flow has the user clicking through a few pages of results of a table, and your table has a pagination component on it like the following:

<x-pagination current-page="1" total-pages="5" ></x-pagination>

The <x-pagination> component is responsible for creating all the pagination buttons that are possible, and highlighting the current page being shown, but the actual buttons the user sees in the UI are created entirely in the shadow DOM with something like this:

// x-pagination.js
render() {
   return html`<ul>
      ${[...Array(this.totalPages - 1).keys()].map((page) => {
         return html`<li><button class="internal-button" @click="emitPageClickEvent()">${page + 1}</button></li>`;
       })}
   </ul>`
}

The [...Array(this.totalPages - 1).keys()] syntax is just a shortcut to create an array with indexes numbered 0 through this.totalPages. And we can assume that this.totalPages has been converted to a number type from its string attribute counterpart.

Let's not trouble ourselves with questions about whether or not this component API is correct, or even if the code in the render function is the right way to create pagination buttons. The main point is that the buttons the user needs to click are generated inside the shadow DOM of <x-pagination>

Now lets say that your test needs to have the user click to page 3 of your dataset because you're testing that the filtering functionality of a larger page is working correctly.

What do you do?

How do you simulate the user going to page 3? You don't own the button that when clicked will emit the event that your code needs to do its "go to page 3 logic" and the component has no api to "change to page 3" because current-page merely shows what page is currently being shown. What I have seen is that people will go digging into the shadow root to grab the element they need then call its .click() method.

That shadow root selector might look something like:

document.getElementByTagName('x-pagination')
   .shadowRoot
   .querySelectorAll('button.internal-button')[2].click();

At first it seems pretty straight-forward. You just query for the parent component, reach into its internals and query for the element you need. You are a dev, you can open Chrome's Dev Tools and Inspect Element like nobody's business, so you know exactly what you're looking for and it all works.

But there's a sneaky problem with this approach, and it'll rear its ugly head when the developer of those components changes the internals of <x-pagination>. And since the internals are private, the developer can change the internal structure WITHOUT a SemVer breaking change release version. So one day soon, your automated dependency refresh pulls in the latest patch version and BOOM, your querySelector is broken, your tests and pipeline fails and you get to go digging to find out that button.internal-button doesn't exist anymore because the developer changed the class for some reason.

So how can this unintentional breach of the SemVer contract be prevented? The component developer should provide a set of first-class internal access properties.

Internal access properties

What is an "Internal Access Property" you ask? Well for starters, it's a term that I just made up when thinking about this problem. I don't really know if there's an industry term for what I'm going to describe, so if there is, please let me know!

Definition
An internal access property is a component property exposed as a first-class API property on a component as a getter that returns either a DOM element or a DOM element collection from the internal component template.

In our <x-pagination> case, a set of internal access properties might look something like:

// x-pagination.js

class XPagination extends LitElement {

   get nextButton() {
      return this.shadowRoot.querySelector('button.next-button');
   }

   get prevButton() {
      return this.shadowRoot.querySelector('button.prev-button');
   }

   get pageButtons() {
      return this.shadowRoot.querySelectorAll('.pagination-container button');
   }

   render() {
      ...
   }
}

What makes internal access properties different from "normal" properties in web components?

• No setter since they are purely for retrieval
• Return a DOM element(s) instead of the usual primitives

From a code perspective, there's not much more to it.

Providing these properties can preemptively solve the unintentional breakage problem (by preventing your consuming developers from having to write brittle test cases) and simplify internal access considerably.

Providing a set of internal access properties with each component gives component consumers an access channel to use internal DOM elements when needed, both for testing and for unforeseen use cases where extension/reconfiguration are needed.

What makes an internal access property different than querying the shadow DOM from the outside?

Without doubt the biggest benefit is testability.

A consuming developer doesn't have an easy mechanism to test to make sure that some internal DOM element still exists for every single test. There's no unit tests that can easily be written by the consuming dev to make sure that all the internals of components they need to access are actually going to exist at test execution time. Also, there's no easy mechanism for devs to verify at dev time either, because their next CICD build could pull in a patch bump of that component package that breaks it.

But the component developer can easily test and guarantee an internal access property. Since they are first-class properties, they would be tested to a) make sure they actually exist and b) verify that they actually return the correct DOM element(s) they are supposed to even when the internal implementation approach changes. When those internals get removed or selectors being used to return those props get changed, the component's unit tests break.

Additionally, changing the internal implementation in a way that removes the need for some internal access property would be a breaking change and would cause a breaking change release.

Recommending that consuming devs use your internal access properties instead of querying the shadow root allows everyone on either end of the development/consumption spectrum to trust in SemVer and allows consuming devs to be actually be able to write tests that aren't brittle.

Won't consuming developers do bad things with DOM elements?

They already had access to the same DOM elements in the first place (with open shadow roots). At the end of the day, we still write javascript. If a developer is willing to write a shadow DOM querySelector they can already get access to component internals. Internal access properties make that process easier for supported use cases. And if/when developers do go mucking around with internal access properties and break things in their applications, component developers would tell them the same thing as if they queried the shadow DOM -- "Sorry but I can't support you since you're not using my approved API for its supported purpose".

A huge reason we make Design Systems and component libraries is to enable developers, not to police them. (thanks Cory) Sure, there will be times that those DOM elements get used for non-supported use cases, but the enablement we get from ensuring tests aren't brittle is WAY more important. As a component library author, the LAST thing I want to do is introduce unexpected breakages in consuming developers' apps OR tests. Internal access properties help cut down on those.

But wait, there's more!

Nested internal access properties

Internal access properties don't always have to just query the immediate component's shadow root elements. They can also be nested so that an internal access property of a parent component returns an internal access property of a child component in complex cases.

Here's an example. Lets say that <x-pagination also has a text input feature (rendered entirely in the shadow root of course) that lets the user filter by some text entered. Lets say the prop that controls that is something like

<x-pagination with-filters ...otherprops></x-pagination>

And when the with-filters boolean attribute is present x-pagination renders an x-input alongside the pagination buttons. And we need to test our filtering mechanism too, so we need to type some value into that input and test what happens to our page. Again, we're not interested in directly testing the internal functionality of x-pagination we're only trying to USE internal things from it to exercise our own tests. So we might do something like this, say using Cypress syntax this time:

cy.get('x-pagination')
   .shadow('x-input')
   .shadow('input')
   .type('My Filter Query');

You might be tempted to just set .value on that input, but simply setting the value prop on a native <input> doesn't trigger any of the events that x-input might be listening to and re-wrapping or re-emitting with custom event names and such, so using something like Cypress' .type() function would be safer because they do some magic to make sure that those events are triggered.

Here we have the same problem as before, but not if there's an internal access property. If x-pagination and x-input have properties like:

class XPagination extends LitElement {

   get nativeInput() {
      return this.shadowRoot.querySelector('x-input').nativeInput;
   }
}

class XInput extends LitElement {

   get nativeInput() {
      return this.shadowRoot.querySelector('input');
   }
}

then, the tester could simply use the nativeInput property on x-pagination and be returned the native <input> from the internal <x-input>'s shadow root.

cy.get('x-pagination')
   .invoke('prop', 'nativeInput').type('some value');

Its a simpler get, there's no explicit shadow DOM querying in the test, and the component developer has tested that nativeInput exists and will return the right native <input> (twice actually, once in x-pagination and once in x-input). And if the component developer decides NOT to use x-input anymore, and updates the nativeInput property in a patch release, the above test doesn't break.

Some testing frameworks require using the native element for interaction

A quick word about testing frameworks is important to mention. Some frameworks like Cypress might require that when you interact with elements, that they are the native ones. So if you are testing a complex component with nested components, you're going to need access to the native <input> or <button> at the end of the component tree so that Cypress' helper functions will work correctly and to avoid errors like

cy.type() failed because it requires a valid typeable element

Thanks Brian for the call out

Nested internal access properties can give testers access to the native elements directly.

Async internal access properties

It's also possible, and probably desirable, to make your internal access properties async as well. The nested case above isn't quite complete, because if the internal x-input component isn't upgraded to a shadow DOM component when the nativeInput property is being retrieved from x-pagination for some reason, then you'd get a null back.

To prevent that, you can make your internal access properties return a Promise that waits for the nested component property to be available. If that nested internal access prop is also async, then you can just await all the way down.

Some web component authoring frameworks have mechanisms to let consumers wait until a component instance has been upgraded (like Lit's await component.updateComplete docs) to do these kinds of retrievals and be sure that shadow DOMs are accessible.

Closed shadow roots

Through this article so far, my comments have largely been made assuming that the shadow roots in the example components were open and accessible from the outside. But when a shadow root is closed, internal access properties become even more important to provide because there is no access from the outside at all.

If the third-party component is created like:

class MyElement extends HTMLElement {
   constructor() {
      super();
      this.root = this.attachShadow({ mode: 'closed' });
   }

   render() {
      return html`<div class="internal">I'm a div in a closed shadow root.</div>`;
   }
}

then trying to access the internal div with

document.querySelector('my-element'.shadowRoot.querySelector('div.internal');

is impossible because the shadowRoot property will be null.

For closed shadow root components, internal access properties are a MUST.

What kinds of internal access properties should be provided?

The best place to start is any element that needs interaction and is created entirely in the shadow DOM. After all, those are the elements most likely to be used in tests.

Think about exposing:

• Buttons
• Form elements
• Anchors

If you have a collection of related items, expose them as a collection to a) limit the number of props on your component, and b) let your consuming dev easily iterate/filter/sort them in tests however they need to.

Some good collections might be:

• Datepicker date selection buttons
• Tab group tab change buttons
• Menu item elements in a menu (if they aren't slotted)

But as always, which internal access properties you expose is going to depend entirely on the kind of component you are creating and how the user will interact with it.

Conclusion

Providing internal access properties as a part of your web component's API can ease the testing burden considerably and prevent random test failures caused when a component's internal implementation changes over time.

I'm sure I've only scratched the surface of the potential use cases for internal access properties, but I do know that once you start looking around for places where you can provide a set of them to your users, you'll find them all over the place. That [x] button in the top corner of your modal window might need clicking, the native input in your input fields might need typing into, internal buttons all over the place.

As always, I'd love to know your thoughts as well. I have looked around various places and haven't seen a topic like this come up, so I'm sure I've missed it and would love some other perspectives on this idea.