What are end-to-end tests?

You can think of automated tests as existing on a spectrum. Although they're all valuable, it's helpful to know the difference between the various types and when to use them.

At one end are unit tests, used to test the functionality of individual methods or components. These are helpful for ensuring that the core building blocks of your applications function as expected in isolation.

In the middle are integration tests. These are tests abstracted up a level from unit tests, executing and asserting on code paths that traverse between multiple units or modules. Now we're starting to ensure the different pieces of our application interact with each other as expected.

Lastly, on the opposite end, are end-to-end (E2E) tests. These are tests aimed to cover the entire surface area of your application from top to bottom. These should generally follow the code paths expected from your end-users to ensure they're as close to reality as possible.

Why should you write end-to-end tests?

In the typical "testing pyramid" (above), E2E tests represent the very top of the pyramid. This is due to the fact that they can be expensive and slow both to write and maintain, and so should make up a smaller percentage of the tests your team is responsible for. What people typically fail to mention around this pyramid, however, is that as you move up the tiers of the pyramid each form of testing is providing you with more confidence.

Why exactly do end-to-end tests provide more confidence? As mentioned before, they should emulate a real user as closely as possible. This means that when one of your E2E tests results in a real failure, you're catching a bug that a real user would have found on their own later. This is much more helpful than a unit test telling me my individual method returns the correct string, or my integration test telling me that my endpoint correctly invokes my method.

And so why, then, is E2E the smallest tier of the testing pyramid? As we touched on above, E2E tests have historically been slow, painful, and expensive for a team to maintain. Frameworks were hard to set up and use, tests were flaky for a variety of common reasons, and often times companies even had manual QA teams to fill this role.

Luckily, things have changed and end-to-end tests are more feasible than ever before.

How do you begin writing end-to-end tests?

Now that we understand what E2E tests are and why they're important, how do we begin to implement them? Let's go over some popular options for web applications today.

For the examples below, let's assume we're testing Amazon's "Add to Cart" functionality. We're going to search for "shoes", add a result to our cart, and simply ensure it's actually in the cart.

Puppeteer (+ Jest)

Puppeteer is a Node library by Google for running a "headless" instance of Chrome. Longstanding and widely known, it's been a large presence in the testing community for some time now. It allows you to programmatically interface with and manipulate Chrome (or Chromium).

Since Puppeteer itself is not intended solely for testing purposes, it must be combined with an existing testing library in order to test and assert as we'd expect (pun intended). Jest is the most popular Javascript option, and can be paired with Puppeteer using jest-puppeteer.

describe('amazon.com', () => {
    beforeAll(async () => {
        await page.goto('https://amazon.com');
    });

    it('should allow adding to cart', async () => {
        const searchInput = await page.$('input#twotabsearchtextbox');
        searchInput.type('shoes');

        const productLink = await page.$('#SEARCH_RESULTS-SEARCH_RESULTS a');
        const productTitle = productLink.evaluate((node) => node.innerText);

        await productLink.click();
        await page.click('#add-to-cart-button');

        await page.click('#hlb-view-cart');
        await page.waitForFunction(`document.querySelector(".sc-product-link").innerText.includes("${productTitle}")`);
    });
});

Selenium (+ Jest)

Selenium WebDriver is another API for automating and interfacing with the browser. As opposed to Puppeteer, Selenium has support for every popular browser.

Like Puppeteer, Selenium itself does not provide testing or assertion functionality. However, we have another way to integrate with Jest: jest-environment-webdriver.

describe('amazon.com', () => {
    beforeAll(async () => {
        await page.goto('https://amazon.com');
    });

    it('should allow adding to cart', () => {
        const searchInput = driver.findElement(By.css('input#twotabsearchtextbox'));
        searchInput.sendKeys('shoes');

        const productLink = driver.findElement(By.css('#SEARCH_RESULTS-SEARCH_RESULTS a'));
        const productName = productLink.getText();

        productLink.click();
        driver.findElement(By.css('#add-to-cart-button')).click();

        driver.findElement(By.css('#hlb-view-cart')).click();
        expect(driver.findElements(By.css('.sc-product-link'))).resolves.toHaveText(productName);
    });
});

cypress.io

Possibly the most popular option today, cypress.io is a Javascript framework and SaaS offering for E2E testing your application. Through their framework and CLI, it's fairly easy to get started writing a simple Javascript test and seeing the results.

describe('amazon.com', () => {
    beforeAll(() => {
        cy.visit('https://amazon.com');
    });

    it('should allow adding to cart', () => {
        const searchInput = cy.get('input#twotabsearchtextbox');
        searchInput.type('shoes');

        const productLink = cy.get('#SEARCH_RESULTS-SEARCH_RESULTS a');
        const productName = productLink.invoke('text');

        productLink.click();
        cy.contains('Add to Cart').click();

        cy.contains('Cart').click();
        cy.contains(productName);
    });
});

walrus.ai

walrus.ai is a new, developer-first SaaS option looking to abstract away the complexities and pain points of true end-to-end testing. Through a combination of AI and human verification, walrus.ai allows developers to specify tests in plain English and with just a single API call.

walrus -a YOUR_API_TOKEN -u https://amazon.com -i \
    'Search for "shoes"' \
    ‘Add a result to your shopping cart’ \
    'Ensure the shoes are in your cart'

Notice how many fewer instructions we need thanks to the use of human-readable sentences? Due to their higher-level description, they're also inherently less fragile than specific selectors.

It's also free to get started with walrus.ai to see if it makes sense for you. Try it out today!

Wrapping up

If you didn't already, hopefully you now understand the difference between the various types of automated tests, as well as the benefits of writing end-to-end tests for your application. Maybe you've even used walrus.ai to get your first E2E test running ASAP!

In a future post, we'll discuss best practices for ensuring your end-to-end tests are both resilient and robust.