DEV Community

Cover image for Testing Web Vitals With Cypress
Craig Morten
Craig Morten

Posted on • Edited on

Testing Web Vitals With Cypress

It is well understood that performance is a critical consideration for any website which can have far reaching impacts on everything from customer satisfaction, SEO, and ultimately your bottom line. You cannot determine the success of performance work without the ability measure the results and compare to performance budgets - this calls for testing infrastructure to make sure you have the necessary visibility on metrics... introducing cypress-web-vitals.

cypress-web-vitals allows you to test against the Google Web Vital signals within your Cypress workflows through a new cy.vitals() custom command.

Web Vitals is an initiative by Google to provide unified guidance for quality signals that are essential to delivering a great user experience on the web.

Reference: https://web.dev/vitals/

 Getting started

In your project, install the dependencies:

npm install --save-dev cypress-web-vitals cypress-real-events
Enter fullscreen mode Exit fullscreen mode

Note: cypress-web-vitals currently makes use of cypress-real-events to click the page to calculate first input delay. Hence it is needed as a peer-dependency.

Within you support commands file (normally cypress/support/commands.js), add this one liner to get setup:

import "cypress-web-vitals";
Enter fullscreen mode Exit fullscreen mode

And now you're set to get going with Web Vital performance budget tests in your Cypress workflow! 🎉

Add you first test like so:

describe("Web Vitals", () => {
  it("should pass the meet Google's 'Good' thresholds", () => {
    cy.vitals({ url: "https://www.google.com/" });
  });
});
Enter fullscreen mode Exit fullscreen mode

You are now set up to test against all of the Web Vitals using Google's "Good" threshold values:

Customise your tests

You can further customise your tests through additional optional configuration which is passed to the cy.vitals(webVitalsConfig) call:

  • Optional url: string - The url to audit. If not provided you will need to have called cy.visit(url) prior to the command.
  • Optional firstInputSelector: string - The element to click for capturing FID. The first matching element is used. Default: "body".
  • Optional thresholds: object - The thresholds to audit the Web Vitals against. If not provided Google's "Good" thresholds will be used. If provided, any missing Web Vitals signals will not be audited.
  • Optional vitalsReportedTimeout: number - Time in ms to wait for Web Vitals to be reported before failing. Default: 10000.

For example:

// Use the `main` element for clicking to capture the FID.
cy.vitals({ firstInputSelector: "main" });

// Test the page against against a CLS threshold of 0.2.
cy.vitals({ thresholds: { cls: 0.2 } });
Enter fullscreen mode Exit fullscreen mode

For more details on usage refer to the API docs.

How does it work?

  1. The url is visited with the HTML response intercepted and modified by Cypress to include the web-vitals script and some code to record the Web Vitals values.
  2. Several elements (if exist) starting with the provided element (based on firstInputSelector) are then clicked in quick succession to simulate a user clicking on the page. Note: if choosing a custom element, don't pick something that will navigate away from the page otherwise the plugin will fail to capture the Web Vitals metrics.
  3. The audit then waits for the page load event to allow for the values of LCP and CLS to settle; which are subject to change as different parts of the page load into view.
  4. Next the audit simulates a page visibility state change which is required for the CLS Web Vital to be reported.
  5. The audit then attempts to wait for any outstanding Web Vitals to be reported for which thresholds have been provided.
  6. Finally the Web Vitals values are compared against the thresholds, logging successful results and throwing an error for any unsuccessful signals. Note: if the audit was unable to record a Web Vital then it is logged, but the test will not fail.

Testing sites in the wild

Here are some example test runs against FAANG homepages to see cypress-web-vitals in action:

Facebook

cy.vitals({ url: "https://www.facebook.com/" });
Enter fullscreen mode Exit fullscreen mode

Cypress Web Vitals test against the Facebook landing page. All metrics below Google's

Amazon

cy.vitals({ url: "https://www.amazon.com/" });
Enter fullscreen mode Exit fullscreen mode

Cypress Web Vitals test against the Amazon home page. All metrics below Google's

Netflix

cy.vitals({
  url: "https://www.netflix.com/gb/",
  firstInputSelector: ".our-story-card-title",
});
Enter fullscreen mode Exit fullscreen mode

Cypress Web Vitals test against the Netflix landing page. All metrics below Google's

For Netflix we have had to introduce a custom element choice for the simulated "first click". Even though first input delay can be measured when in cases where there is no event listener registered to the element, there are scenarios where clicking the body doesn't cut it. Some good examples of elements that will reliably trigger an FID metric are:

  • Text fields, checkboxes, and radio buttons (<input>, <textarea>)
  • Select dropdowns (<select>)
  • Links (<a>)

In order to ensure the Web Vitals can be tested against, it is best to try find an element that reliably triggers an FID, but won't navigate away from the page (perhaps avoid <a>!).

 Google

cy.vitals({ url: "https://www.google.com/" });
Enter fullscreen mode Exit fullscreen mode

Cypress Web Vitals test against the Google home page. All metrics below Google's

Wrap-up

Using any awesome performance testing tooling lately? Tried out cypress-web-vitals on your site and have results to share? Got any comments, queries, or questions? Leave a comment below!

That's all folks! 🚀

Top comments (6)

Collapse
 
ir3ne profile image
irène

Hey @craigmorten thanks for the article!

Using cypress-web-vitals I noticed the results of a page are different compared to results I have running Lighthouse from the browser for the same page. Did this happen to you too? Thanks

Collapse
 
craigmorten profile image
Craig Morten • Edited

Hey @ir3ne 👋

Yes I would expect the results to be different. You will encounter some natural variation regardless (even between runs on the same page with lighthouse) due to volatility in network etc., but beyond that I’d generally expect the results to be slower reported by this package than by Lighthouse.

With this package we’re running the page instrumented with Cypress which will have a performance impact compared with just loading the page in the browser by itself. It also is injecting the web-vitals snippet into the page to get the results which suffers from the Observer Effect of introducing extra page weight (though minimised best as possible). Lighthouse (assuming devtools) doesn’t suffer from these points so likely gives a more accurate reading of performance (unless using lighthouse and Cypress, in which case it depends).

If you want your “lab” results to be as close to real user performance then lighthouse is likely better (though with the caveat that even lighthouse isn’t realistic, only real user monitoring can give a real gauge!). If using for relative performance budgeting/benchmarks then either is hopefully fine 🙃

Collapse
 
ir3ne profile image
irène

@craigmorten thanks for your exhaustive reply. I appreciate that :)

Collapse
 
priteshusadadiya profile image
Pritesh Usadadiya

This article was curated as a part of #52nd Issue of Software Testing Notes newsletter.

Collapse
 
giloagilo profile image
gilo-agilo

Hi @craigmorten ,
Google start analyzing new metric: INP.
Do you have any plans on adding it to cypress-web-vitals ?

Collapse
 
craigmorten profile image
Craig Morten • Edited

There is an issue open and have some rough ideas on what to do, but haven’t got around to it.

github.com/cmorten/cypress-web-vit...

Hopefully will find some time soon 😊