Jordan Mocks Puppeteer with Jest

Demo code here

Go ahead right now and google “Unit testing Puppeteer scripts”. Do it. The results…are good. If you are trying to use Puppeteer to test your product.

But what if your product is a Puppeteer script? I have searched long and hard and haven’t been able to find a good solution. And this is a big problem for someone like me who loves to have good unit tests and loves to use Puppeteer.

So…the purpose of this post is to show how I unit test Puppeteer scripts using Jest. The test framework isn’t overly important but this post makes a lot more sense for those of you who are usign Jest for your unit tests. If you aren’t familiar with Puppeteer, I’d recommend my guide for getting starting with web scraping using Puppeteer. Of course, I don’t imagine many of you will be reading this post if you don’t use Puppeteer.

Getting started

I created a simple function that I could test. While this isn’t as big or complex as a lot of the things Puppeteer is used for, it does showcase most of the key functionalities and goes pretty deep into the Puppeteer module.

export async function action() {
    const browser = await puppeteer.launch({ headless: false });
    const page = await browser.newPage();

    const url = 'https://javascriptwebscrapingguy.com';

    await page.goto(url)

    const entryTitlesHandles = await page.$$('h2.entry-title');

    const links: any[] = [];

    for (let i = 0; i < entryTitlesHandles.length; i++) {
        const link = await entryTitlesHandles[i].$eval('a', element => element.getAttribute('href'));

        links.push(link);
    }

    await browser.close();

    return links;

}

I navigate to javascriptwebscrapingguy, get all blog posts, and then pluck the href out of the element of each one. This way I have to mock puppeteer.launch, browser.newPage, page.goto, page.$$, elementHandle.$eval (though $eval also exists on the page method), and browser.close.

I’ve never mocked anything that deep before. puppeteer.launch returns a Browser, which has a method that returns a Page, which has a method that returns either an ElementHandle (or an array of them).

The mock

Here’s the mock itself:

import { Browser, Page, ElementHandle } from "puppeteer";

export const stubPuppeteer = {
    launch() {
        return Promise.resolve(stubBrowser);
    }
} as unknown as any;

export const stubBrowser = {
    newPage() {
        return Promise.resolve(stubPage);
    },
    close() {
        return Promise.resolve();
    }
} as unknown as Browser;

export const stubPage = {
    goto(url: string) {
        return Promise.resolve();
    },
    $$(selector: string): Promise<ElementHandle[]> {
        return Promise.resolve([]);
    },
    $(selector: string) {
        return Promise.resolve(stubElementHandle);
    },
    $eval(selector: string, pageFunction: any) {
        return Promise.resolve();
    }
} as unknown as Page;

export const stubElementHandle = {
    $eval() {
        return Promise.resolve();
    }
} as unknown as ElementHandle;

This goes through all of the things I use in the test and fully mocks them out. You can see that from top to bottom, it provides the stubbed methods which include the stubbed methods that that stubbed method provides. Me writing it makes it sound terribly confusing. Hopefully seeing it above is more helpful.

The tests

To start, this was the part that was most difficult for me to understand or get right. Jest is pretty great for testing and can allow you to just automock modules by just going jest.mock('moduleName').

That’s pretty powerful but for me, unless there is some voodoo I don’t know about, it wouldn’t handle deep modules such as Puppeteer. This makes sense, because how could it know what you want the deeper methods to return or not return. You are able to provide your mock for the module, however, like this:

jest.mock('puppeteer', () => ({
    launch() {
        return stubBrowser;
    }
}));

And…this provides the rest. I really tried to just return stubPuppeteer directly but I could not figure out why it wouldn’t work. I may mess around it with more for next week’s post. It just throws the following error any time I try:

In any case, doing it this way, returning the manual mock for puppeteer, it provides all the methods needed. All of the tests are shown in the demo code but I would like to discuss some of the more tricky ones here.

This section of code was the most complicated in my opinion:

    const entryTitlesHandles = await page.$$('h2.entry-title');

    const links: any[] = [];

    for (let i = 0; i < entryTitlesHandles.length; i++) {
        const link = await entryTitlesHandles[i].$eval('a', element => element.getAttribute('href'));

        links.push(link);
    }

I get the ElementHandles and then I loop through them, calling $eval and getting the href attribute. So I tested it with just a single link and then with two.

    test('that it should return an array with a single link', async () => {
        jest.spyOn(stubPage, '$$').mockReturnValue(Promise.resolve([stubElementHandle]));
        jest.spyOn(stubElementHandle, '$eval').mockReturnValue(Promise.resolve('https://pizza.com'));

        const result = await action();

        expect(result).toEqual(['https://pizza.com']);
    });

    test('that it should return an array with multiple links', async () => {
        jest.spyOn(stubPage, '$$').mockReturnValue(Promise.resolve([stubElementHandle, stubElementHandle]));
        const stubElementHandleSpy = jest.spyOn(stubElementHandle, '$eval')
            .mockReturnValueOnce(Promise.resolve('https://pizza.com'))
            .mockReturnValueOnce(Promise.resolve('https://github.com'));

        const result = await action();

        expect(result).toEqual(['https://pizza.com', 'https://github.com']);
        expect(stubElementHandleSpy).toHaveBeenCalledTimes(2);
    });

Using Jest’s spyOn and mockReturnValue, I was able to easily return the value I wanted to for each of those functions. When I wanted to handle an array, I just used mockReturnValueOnce and then daisy chained them where they would return one value the first time the function was called and then second value the second time it was called.

Honestly, it all worked really great and was simple. The mock was trickiest part. After that, it was unit testing as usual. I had a good time.

The end.

Demo code here

Looking for business leads?

Using the techniques talked about here at javascriptwebscrapingguy.com, we’ve been able to launch a way to access awesome business leads. Learn more at Cobalt Intelligence!

The post Jordan Mocks Puppeteer with Jest appeared first on JavaScript Web Scraping Guy.