Container Queries: Cross-Resolution Testing

Originally published on crunchingnumbers.live

Failure to test is why I started to work on ember-container-query.

A few months ago, my team and I introduced ember-fill-up to our apps. It worked well but we noticed something strange. Percy snapshots that were taken at a mobile width would show ember-fill-up using the desktop breakpoint. They didn't match what we were seeing on our browsers.

For a while, we ignored this issue because our CSS wasn't great. We performed some tricks with flex and position that could have affected Percy snapshots. Guess what happened when we switched to grid and improved the document flow. We still saw incorrect Percy snapshots.

1. Whodunit? ember-qunit.

To eliminate ember-fill-up as a suspect, I used modifiers to recreate the addon. To my surprise and distress, using modifiers didn't fix the problem. After many trial-and-errors, I found the culprit: ember-qunit.

By default, ember-qunit scales the test window so that your app fits inside its test container.

#ember-testing {
  width: 200%;
  height: 200%;
  transform: scale(0.5);
  transform-origin: top left;
}

What does this mean? When you write tests, you cannot trust DOM render decisions that are based on width or height. Decisions made by media queries and addons like ember-container-query, ember-fill-up, ember-responsive, and ember-screen. Because what your test saw differed from what you saw on a browser, you might have had to mock a service (fake the window size) to get certain elements to (dis)appear.

Fortunately, there is an escape hatch. We can apply the .full-screen class to the test container (#ember-testing-container) to undo the scaling.

.full-screen #ember-testing {
  position: absolute;
  width: 100%;
  height: 100%;
  transform: scale(1);
}

As an aside, this class is applied when we enable development mode, a relatively unknown feature.

In my opinion and conjecture, we (Ember community) didn't really notice this problem and fix it because we were used to writing tests only at 1 resolution: the 1440 × 900 px desktop. We are also prone to designing the web for desktop first. Had we been able to test multiple resolutions easily, I think today's state of testing would be even better.

2. Cross-Resolution Testing

How can you test your app and addon at multiple resolutions?

We need to be able to mark tests that can only be run at one resolution. After all, there will be user workflows that make sense only on mobile or tablet, for example. My team and I followed the Octane craze and introduced filters that look like decorators:

// Mobile only
test('@mobile A user can do X in Dashboard');

// Tablet only
test('@tablet A user can do X in Dashboard');

// Any resolution
test('A user can do X in Dashboard');

Let's go over how to update your test setup, configure CI, and write a Percy test helper to allow these filters.

I'll use GitHub Actions for CI. Describing each line of code can get boring so I'll convey the idea and simplify code in many cases. I encourage you to look at ember-container-query to study details and use my latest code.

a. testem.js

We'll start by updating testem.js. It is responsible for setting the window size.

The idea is to dynamically set the window size based on an environment variable. I'll call this variable DEVICE.

const FILTERS = {
  mobile: '/^(?=(.*Acceptance))(?!(.*@tablet|.*@desktop))/',
  tablet: '/^(?=(.*Acceptance))(?!(.*@mobile|.*@desktop))/',
  desktop: '/^(?!(.*@mobile|.*@tablet))/'
};

const WINDOW_SIZES = {
  mobile: '400,900',
  tablet: '900,900',
  desktop: '1400,900'
};

const { DEVICE = 'desktop' } = process.env;

const filter = encodeURIComponent(FILTERS[DEVICE]);
const windowSize = WINDOW_SIZES[DEVICE];
const [width, height] = windowSize.split(',');

module.exports = {
  test_page: `tests/index.html?filter=${filter}&width=${width}&height=${height}`,
  browser_args: {
    Chrome: {
      ci: [
        `--window-size=${windowSize}`
      ]
    }
  }
};

From lines 15-16, we see that DEVICE decides how tests are run. In QUnit, we can use regular expressions to filter tests. I used lookaheads to say, "When DEVICE=mobile, only run application tests with @mobile filter or application tests without any filter." I decided to run rendering and unit tests only when DEVICE=desktop because they are likely independent of window size.

On line 20, the query parameters width and height are extra and have an important role. I'll explain why they are needed when we write the test helper for Percy.

b. Reset Viewport

Next, we need to apply the .full-screen class to the test container.

There are two options. We can create a test helper if there are few application test files (likely for an addon), or an initializer if we have many (likely for an app).

// Test helper
export default function resetViewport(hooks) {
  hooks.beforeEach(function() {
    let testingContainer = document.getElementById('ember-testing-container');
    testingContainer.classList.add('full-screen');
  });
}


// Initializer
import config from 'my-app-name/config/environment';

export function initialize() {
  if (config.environment === 'test') {
    let testingContainer = document.getElementById('ember-testing-container');
    testingContainer.classList.add('full-screen');
  }
}

export default {
  initialize
}

c. package.json

The last step for an MVP (minimum viable product) is to update the test scripts.

Since Ember 3.17, npm-run-all has been available to run scripts in parallel. I'll assume that you also have ember-exam and @percy/ember.

{
  "scripts": {
    "test": "npm-run-all --parallel test:*",
    "test:desktop": "percy exec -- ember exam --test-port=7357",
    "test:mobile": "DEVICE=mobile percy exec -- ember exam --test-port=7358",
    "test:tablet": "DEVICE=tablet percy exec -- ember exam --test-port=7359"
  }
}

In addition to setting DEVICE, it's crucial to use different port numbers. Now, we can run yarn test to check our app at 3 window sizes. If you have disparate amounts of tests for desktop, mobile, and tablet, you can set different --split values so that you allocate more partitions to one window size. For example, 4 partitions to desktop, 2 to mobile, and 1 to tablet.

d. CI

Your code change may depend on what features your CI provider offers and how many ember-exam partitions you used to test a window size. I don't know what your CI looks like right now, so I'll make a handwave.

In ember-container-query, I didn't split tests into multiple partitions. There simply weren't that many. As a result, I was able to use matrix to simplify the workflow:

jobs:
  test-addon:
    strategy:
      matrix:
        device: [desktop, mobile, tablet]
    steps:
      - name: Test
        uses: percy/exec-action@v0.3.0
        run:
          custom-command: yarn test:${{ matrix.device }}

e. Test Helper for Percy

The end is the beginning is the end. We want to write a test helper for Percy, the thing that launched me into a journey of discovery.

In its simplest form, the test helper understands filters and knows the window size. It also generates a unique snapshot name that is human readable.

import percySnapshot from '@percy/ember';

export default async function takeSnapshot(qunitAssert) {
  const name = getName(qunitAssert);
  const { height, width } = getWindowSize();

  await percySnapshot(name, {
    widths: [width],
    minHeight: height
  });
}

function getName(qunitAssert) { ... }

function getWindowSize() {
  const queryParams = new URLSearchParams(window.location.search);

  return {
    height: Number(queryParams.get('height')),
    width: Number(queryParams.get('width'))
  };
}

On line 13, I hid implementation details. The idea is to transform QUnit's assert object into a string.

Line 16 is the interesting bit. Earlier, when we updated testem.js, I mentioned passing width and height as query parameters. I tried two other approaches before.

In my first attempt, I stored process.env.DEVICE in config/environment.js and imported the file to the test helper file. From WINDOW_SIZES, one can find out width and height from DEVICE. For QUnit, this worked. For Percy, it did not. Since v2.x, Percy doesn't hook into the Ember build pipeline so DEVICE was undefined.

In my second attempt, I used window.innerWidth and window.innerHeight to get direct measurements. innerWidth gave the correct width, but innerHeight turned out to be unreliable. Because I wanted to test at multiple widths and multiple heights, I rejected this approach as well.

3. How to Run Tests

After we make these changes, an important question remains. How do we run tests locally?

• yarn test to run all desktop, mobile, and tablet tests in parallel
• yarn test:desktop --server to run all desktop tests with --server option
• DEVICE=mobile ember test --filter='@mobile A user can do X in Dashboard' to run a particular test

4. What's Next?

On the long horizon, I'd like us to reexamine and change why we are currently limited to testing 1 resolution. Ember's testing story is an already amazing one. I believe the ability to test multiple resolutions (and do so easily without taking 5 steps like above) would make that story even better.

For nearer goals, I'd like us to iron out a couple of issues in overriding ember-qunit:

• Even with .full-screen, the test container's height can be off if we use --server to launch the test browser. If assertions sometimes fail due to incorrect window size, it's harder to separate true and false positives.
• Visiting localhost:4200/tests to start tests will also throw off the test container's width and height. It may be impractical to ask developers to run tests with --server because this does not launch Ember Inspector.

We need to look at allowing cross-resolution testing for ember-mocha as well.

5. Notes

Special thanks to my team Sean Massa, Trek Glowacki, and Saf Suleman for trying out a dangerously unproven, new testing approach with me.