Juro Oravec

Posted on Aug 30, 2023 • Edited on Sep 1, 2023

Portadom: A Unified Interface for DOM Manipulation

#webscraping #javascript #html #webdev

Introduction

Web scraping, while immensely useful, often requires developers to navigate a sea of tools and libraries, each with its own quirks and intricacies. Whether it's JSDOM, Cheerio, Playwright, or even just plain old vanilla JS in the DevTools console, moving between these platforms can be a challenge.

Enter Portadom, your new best friend in the world of web scraping.

What is Portadom?
Portadom provides a consistent DOM manipulation interface across:

Browser API
JSDOM
Cheerio
Playwright

This means you no longer have to rewrite or refactor large chunks of your code when switching between these tools. Instead, you can focus on the logic of your web scraping tasks and let Portadom handle the DOM manipulation intricacies.

The Portadom Workflow

Imagine you're working on a project to scrape data from several websites. You initially start by prototyping in the DevTools console using vanilla JS. Once you've figured out the transformations and data extractions, you realize that some sites can be scraped with static HTML, while others need a JS runtime.

1. Prototyping with Vanilla JS

You start with a simple site and define your transformations directly in the DevTools:

let title = document.querySelector('h1').innerText;

2. Static HTML with JSDOM or Cheerio

For sites where static HTML is sufficient, you can easily migrate your vanilla JS logic:

import { load as loadCheerio } from 'cheerio';
import { cheerioPortadom } from 'portadom';

const html = `<h1>Welcome to Portadom</h1>`;
const $ = loadCheerio(html);
const dom = cheerioPortadom($.root(), null);

const title = await dom.findOne('h1').text();

With Portadom, the transition feels almost seamless. The core logic remains consistent, and only the setup changes.

3. Dynamic Sites with Playwright

For websites that rely heavily on JavaScript, you'd need a tool like Playwright. But with Portadom, even this transition is smooth:

import { playwrightLocatorPortadom } from 'portadom';

const page = await browser.newPage();
await page.goto('https://example.com');

const bodyLoc = page.locator('body');
const dom = playwrightLocatorPortadom(bodyLoc, page);

const title = await dom.findOne('h1').text();

Notice how, once again, only the setup changed. The actual DOM querying logic remains consistent, thanks to Portadom.

Embracing Flexibility

Portadom is all about flexibility. No matter where you start — be it with Cheerio for static HTML parsing or Playwright for dynamic sites — you're never locked in. If your needs change, Portadom makes it easy to switch your underlying platform without overhauling your entire scraping logic.

Take the Leap with Portadom

Web scraping is finicky - everything breaks all the time. With Portadom, you're equipped with a tool that lets you focus on crafting the perfect data extraction strategy without getting bogged down by the intricacies of various DOM manipulation libraries. Dive in and let Portadom streamline your web scraping journey!

Portadom was already successfully used in scraping:

Portadom currently supports following manipulations:

Element attributes, properties, text
findOne - equivalent to document.querySelector
findMany - equivalent to document.querySelectorAll
closest - equivalent to Element.closest
parent - equivalent to Element.parent
children - equivalent to Element.children
root - Get document root
remove - Remove current Element
getCommonAncestor - Get common ancestor between this and other Element
getCommonAncestorFromSelector - Get common ancestor between this and other Element (found by selector)

Chaining

For cross-compatibility, each method on a Portadom instance returns
a Promise.

But this then leads to then / await hell when you need to call multiple methods in a row:

const employerName = (await (await el.findOne('.employer'))?.text()) ?? null;

To get around that, the results are wrapped in chainable instance. This applies to each method that returns a Portadom instance, or an array of Portadom instances.

So instead, we can call:

const employerName = await el.findOne('.employer').text();

You don't have to chain the commands. Instead, you can access the associated promise under promise property. For example this:

const mapPromises = await dom.findOne('ul')
  .parent()
  .findMany('li[data-id]')
  .map((li) => li.attr('data-id'));
const attrs = await Promise.all(mapResult);

Is the same as:

const ul = await dom.findOne('ul').promise;
const parent = await ul?.parent().promise;
const idEls = await parent?.findMany('li[data-id]').promise;
const mapPromises = idEls?.map((li) => li.attr('data-id')) ?? [];
const attrs = await Promise.all(mapPromises);

Examples

Example - Profesia.sk

See source code here.

// Following lines added for completeness 
const $ = loadCheerio(html);
const dom = cheerioPortadom($.root(), url);
// ...
const rootEl = dom.root();
const url = await dom.url();

// Find and extract data
const entries = await rootEl.findMany('.list-row:not(.native-agent):not(.reach-list)')
  .mapAsyncSerial(async (el) => {
  const employerName = await el.findOne('.employer').text();
  const employerUrl = await el.findOne('.offer-company-logo-link').href();
  const employerLogoUrl = await el.findOne('.offer-company-logo-link img').src();

  const offerUrlEl = el.findOne('h2 a');
  const offerUrl = await offerUrlEl.href();
  const offerName = await offerUrlEl.text();
  const offerId = offerUrl?.match(/O\d{2,}/)?.[0] ?? null;

  const location = await el.findOne('.job-location').text();

  const salaryText = await el.findOne('.label-group > a[data-dimension7="Salary label"]').text();

  const labels = await el.findMany('.label-group > a:not([data-dimension7="Salary label"])')
    .mapAsyncSerial((el) => el.text())
    .then((arr) => arr.filter(Boolean) as string[]);

  const footerInfoEl = el.findOne('.list-footer .info');
  const lastChangeRelativeTimeEl = footerInfoEl.findOne('strong');
  const lastChangeRelativeTime = await lastChangeRelativeTimeEl.text();
  // Remove the element so it's easier to get the text content
  await lastChangeRelativeTimeEl.remove();
  const lastChangeTypeText = await footerInfoEl.textAsLower();
  const lastChangeType = lastChangeTypeText === 'pridané' ? 'added' : 'modified';

  return {
    listingUrl: url,
    employerName,
    employerUrl,
    employerLogoUrl,
    offerName,
    offerUrl,
    offerId,
    location,
    labels,
    lastChangeRelativeTime,
    lastChangeType,
  };
});

Example - SKCRIS

See source code here.

// Following lines added for completeness. Edited for brevity.
const $ = loadCheerio(html);
const dom = cheerioPortadom($.root(), url);
// ...
const url = await dom.url();
const rootEl = dom.root();
const tableDataEls = await rootEl
  .findMany('.detail > tr')
  .filterAsyncSerial((el) => el.text()) // Remove empty tags
  .slice(1, -1).promise; // Remove first row (heading) and last row (related resources)

const tableData = tableDataEls.reduce(async (promiseAgg, rowEl) => {
  const agg = await promiseAgg;
  const [title, val] = await rowEl.children()
    .mapAsyncSerial(async (el) => {
      const text = await el.text();
      return text?.replace(/\s+/g, ' ') ?? null;
    });
  if (!title) return agg;

  agg[title] = val ?? null;
  return agg;
}, Promise.resolve({} as Record<string, string | null>));

return tableData;

Example - Facebook post timestamp

Facebook prompted the need to add the getCommonAncestor method, as Facebook's HTML doesn't provide many reliable patterns to work with.

Note how we used getCommonAncestor to get an element that wasn't easily targettable by class/attribute selectors.

// Following lines added for completeness. Edited for brevity.
const body = await page.evaluateHandle(() => document.body)
const dom = playwrightHandlePortadom(body, page);
// ...
// Find container with post stats
const likesEl = await dom.findOne('[aria-label*="Like:"]').promise;
const commentsEl = await dom
  .findMany('[role="button"] [dir="auto"]')
  .findAsyncSerial(async (el) => {
    const text = await el.text();
    return text?.match(URL_REGEX.COMMENT_COUNT);
  }).promise;

const statsContainerEl =
  likesEl?.node && commentsEl?.node
    ? await likesEl.getCommonAncestor(commentsEl.node).promise
    : null;
// "6.9K views"
const viewsText = await statsContainerEl
  ?.children()
  .findAsyncSerial(async (domEl) => {
    const text = await domEl.text();
    return text?.match(/views/i);
  })
  .textAsLower();

Learn more

Top comments (5)

Tommy Chan • Aug 31 '23

There are multiple way to reduce lodash size.

Most user should be using ES Modules in these days. Switching to lodash-es and let the bundler to do tree shaking.
Use babel-plugin-lodash
Bundle / Minify this library with a proper setup bundler.

Also this is a Web Scraping library. The source code should already be there and thus no network traffic time. Isn't it micro-optimize?