Finding Repeated og:title and og:description Across a Whole Site From Its sitemap.xml

Checking one page's share tags is easy. You paste the URL into a checker, you read the title and the description, you fix what looks wrong. The problem I keep running into on client sites is not one bad page. It is forty pages that are each fine on their own and identical to each other.

A template sets og:description to the site tagline. A product feed copies the product name into the description field. A blog theme falls back to the same sentence whenever the excerpt is empty. Every one of those pages passes a single-URL check, because the tags exist and the image loads. But paste three of them into a team chat and you get three cards that say the same thing, and nobody can tell which link is which.

This post builds a small Node script that starts from sitemap.xml, fetches every listed page, reads the share tags from the <head>, and prints groups of URLs that share an og:title, share an og:description, or use a description that just repeats the title. It has no dependencies, it follows sitemap index files, and it comes with a node:test suite that runs against a local HTTP server.

Why a sitemap is the right starting list

You could crawl the site by following links, but a crawler has to make decisions you do not care about here: how deep to go, which query strings matter, what to do with pagination. The sitemap already answers the question "which URLs do we want people to find and share?" If a page is in the sitemap, someone expects it to be visited, and a visited page gets shared.

It also keeps the script honest. If the sitemap is broken, the duplicate report is meaningless, so it is worth confirming the file first. I run the sitemap through a sitemap validator before I trust any list built from it: it catches <loc> values that are only a path such as /about, a wrong root element or namespace, and files that are really an HTML error page served with a 200.

Two shapes of sitemap show up in practice:

• A <urlset> with <url><loc>…</loc></url> entries. Small sites usually have one of these.
• A <sitemapindex> with <sitemap><loc>…</loc></sitemap> entries pointing at more sitemaps. WordPress, many CMSs and most generators split large sites this way.

The script handles both, and it decodes entities in <loc>, because & in a query string is normal and correct inside XML.

What counts as "the same"

Exact string comparison misses most real duplicates. One template writes Catering & Events. and another writes catering & events in a different place, and they are the same line to a reader. So before comparing, the script normalises each value:

• decode HTML entities (&, ", numeric references)
• Unicode NFKC normalisation, so full-width and compatibility characters compare equal
• lower-case
• collapse runs of whitespace
• strip trailing punctuation such as ., !, … and dashes

It does not try fuzzy matching. Two descriptions that differ by one word are a judgement call, and a script that guesses will either bury you in false positives or miss the boring exact copies that make up most of the problem. Exact-after-normalising is the useful first pass.

There are three findings:

1. Shared og:title: two or more URLs send the same headline.
2. Shared og:description: two or more URLs send the same summary line. This is the common one.
3. Description repeats title: on a single URL, the summary line adds nothing because it is the headline again. The guide on og:description that repeats og:title explains why that wastes the second line of the card, especially in chat bubbles that only keep a few words.

Reading tags without a DOM

The script only needs a handful of values from the head, so it uses small regular expressions rather than a parser. That is a deliberate trade-off for a reporting tool: the input is HTML your own site produced, and the job is to read meta tags, not to understand the page. A few details matter:

• Only the <head> is read. Some page builders print a stray <meta property="og:title"> inside the body, and crawlers that respect document structure ignore it.
• Attribute order varies. <meta content="…" property="og:description"> is just as valid as the usual order.
• The first tag wins when a key appears twice. Duplicate tags are their own bug, but for grouping purposes I record what most parsers will pick up first.
• Single and double quotes are both accepted around attribute values.

The script

Here is the whole module. It exports the pieces so the test file can use them, and it runs as a CLI when executed directly.

// share-dupes.mjs — find repeated og:title / og:description across the URLs in a sitemap.
// Node 20+ (global fetch). Usage: node share-dupes.mjs https://example.com/sitemap.xml

const ENTITIES = { amp: "&", lt: "<", gt: ">", quot: '"', apos: "'", nbsp: " " };
export function decode(s) {
  return s.replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (m, e) => {
    if (e[0] === "#") return String.fromCodePoint(e[1].toLowerCase() === "x" ? parseInt(e.slice(2), 16) : Number(e.slice(1)));
    return ENTITIES[e.toLowerCase()] ?? m;
  });
}

export function sitemapLocs(xml) {
  const isIndex = /<sitemapindex[\s>]/i.test(xml);
  const locs = [...xml.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/gi)].map((m) => decode(m[1]));
  return { isIndex, locs };
}

function attr(tag, name) {
  const m = new RegExp(`\\b${name}\\s*=\\s*("([^"]*)"|'([^']*)')`, "i").exec(tag);
  return m ? (m[2] ?? m[3]) : undefined;
}

export function shareTags(html) {
  const head = html.split(/<\/head>/i)[0];
  const out = {};
  for (const [tag] of head.matchAll(/<meta\b[^>]*>/gi)) {
    const key = (attr(tag, "property") ?? attr(tag, "name"))?.toLowerCase();
    const content = attr(tag, "content");
    if (!key || content === undefined || key in out) continue; // first tag wins
    if (["og:title", "og:description", "description"].includes(key)) out[key] = decode(content).trim();
  }
  const t = /<title[^>]*>([\s\S]*?)<\/title>/i.exec(head);
  if (t) out.title = decode(t[1]).trim();
  return out;
}

export function normalise(s) {
  return (s ?? "").normalize("NFKC").toLowerCase().replace(/\s+/g, " ").replace(/[\s.!?…:;,–—-]+$/u, "").trim();
}

export function findRepeats(pages) {
  const groups = { "og:title": new Map(), "og:description": new Map() };
  const sameAsTitle = [];
  const missing = [];
  for (const p of pages) {
    for (const key of Object.keys(groups)) {
      const v = normalise(p.tags[key]);
      if (!v) { missing.push({ url: p.url, key }); continue; }
      if (!groups[key].has(v)) groups[key].set(v, { text: p.tags[key], urls: [] });
      groups[key].get(v).urls.push(p.url);
    }
    const t = normalise(p.tags["og:title"]), d = normalise(p.tags["og:description"]);
    if (t && d && t === d) sameAsTitle.push(p.url);
  }
  const repeats = {};
  for (const [key, m] of Object.entries(groups)) {
    repeats[key] = [...m.values()].filter((g) => g.urls.length > 1).sort((a, b) => b.urls.length - a.urls.length);
  }
  return { repeats, sameAsTitle, missing };
}

async function getText(url, fetchImpl) {
  const res = await fetchImpl(url, { redirect: "follow", headers: { accept: "text/html,application/xml;q=0.9" } });
  if (!res.ok) throw new Error(`${res.status} for ${url}`);
  return res.text();
}

export async function collect(sitemapUrl, { fetchImpl = fetch, concurrency = 4, limit = 500 } = {}) {
  const urls = [];
  const queue = [sitemapUrl];
  while (queue.length && urls.length < limit) {
    const { isIndex, locs } = sitemapLocs(await getText(queue.shift(), fetchImpl));
    if (isIndex) queue.push(...locs);
    else urls.push(...locs);
  }
  const todo = urls.slice(0, limit);
  const pages = [], errors = [];
  async function worker() {
    while (todo.length) {
      const url = todo.shift();
      try { pages.push({ url, tags: shareTags(await getText(url, fetchImpl)) }); }
      catch (e) { errors.push({ url, error: e.message }); }
    }
  }
  await Promise.all(Array.from({ length: concurrency }, worker));
  return { pages, errors };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const { pages, errors } = await collect(process.argv[2]);
  const { repeats, sameAsTitle, missing } = findRepeats(pages);
  console.log(`checked ${pages.length} pages, ${errors.length} fetch errors`);
  for (const [key, groups] of Object.entries(repeats)) {
    for (const g of groups) console.log(`\n${key} used on ${g.urls.length} pages: "${g.text}"\n  ${g.urls.join("\n  ")}`);
  }
  if (sameAsTitle.length) console.log(`\nog:description repeats og:title:\n  ${sameAsTitle.join("\n  ")}`);
  if (missing.length) console.log(`\nmissing:\n  ${missing.map((m) => `${m.key} ${m.url}`).join("\n  ")}`);
  for (const e of errors) console.log(`error: ${e.url} ${e.error}`);
  process.exitCode = Object.values(repeats).some((g) => g.length) || sameAsTitle.length ? 1 : 0;
}

A few choices worth calling out:

Concurrency is small on purpose. Four workers is enough to finish a few hundred URLs quickly without looking like an attack on your own server. If the site is behind a WAF or a rate limit, drop it to one or two.

There is a hard limit. limit defaults to 500 URLs. A large shop with tens of thousands of product URLs does not need every product checked to discover that the template copies the product name into the description. Sample, fix the template, run again.

Fetch errors are reported, not fatal. A 404 in the sitemap is useful information (that page should not be listed), but it should not stop the duplicate report.

It exits non-zero when it finds repeats. That makes it usable in CI after a deploy, if you want it there. I mostly run it by hand after a template change.

Testing it against a fake site

The tests spin up a local HTTP server with a sitemap index, one child sitemap, three pages and one URL that returns 404. The pages are written to trigger each finding: two pages share a description, one page's description is its title with different case and punctuation.

import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { createServer } from "node:http";
import { sitemapLocs, shareTags, normalise, findRepeats, collect } from "./share-dupes.mjs";

const page = (title, desc) => `<!doctype html><html><head><title>${title}</title>
<meta property="og:title" content="${title}"><meta content="${desc}" property="og:description"></head><body><meta property="og:title" content="body tag ignored"></body></html>`;

let server, base;
before(async () => {
  server = createServer((req, res) => {
    const routes = {
      "/sitemap-index.xml": `<?xml version="1.0"?><sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>${base}/sitemap-posts.xml</loc></sitemap></sitemapindex>`,
      "/sitemap-posts.xml": `<?xml version="1.0"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        <url><loc>${base}/a</loc></url><url><loc>${base}/b</loc></url><url><loc>${base}/c</loc></url><url><loc>${base}/gone</loc></url></urlset>`,
      "/a": page("Spring menu", "Our spring menu, with three new dishes."),
      "/b": page("Opening hours", "Our spring menu, with three new dishes."),
      "/c": page("Catering &amp; events", "Catering &amp; Events."),
    };
    const body = routes[req.url];
    res.writeHead(body ? 200 : 404, { "content-type": req.url.endsWith(".xml") ? "application/xml" : "text/html" });
    res.end(body ?? "not found");
  });
  await new Promise((r) => server.listen(0, "127.0.0.1", r));
  base = `http://127.0.0.1:${server.address().port}`;
});
after(() => server.close());

test("reads a sitemap index and a urlset", () => {
  assert.equal(sitemapLocs("<sitemapindex><sitemap><loc> https://x.dev/s.xml </loc></sitemap></sitemapindex>").isIndex, true);
  assert.deepEqual(sitemapLocs("<urlset><url><loc>https://x.dev/a?x=1&amp;y=2</loc></url></urlset>").locs, ["https://x.dev/a?x=1&y=2"]);
});

test("takes the first og:title in the head, any attribute order", () => {
  const t = shareTags(page("Fish &amp; Chips", "Fried, with &quot;mushy&quot; peas"));
  assert.equal(t["og:title"], "Fish & Chips");
  assert.equal(t["og:description"], 'Fried, with "mushy" peas');
});

test("normalise ignores case, spacing and trailing punctuation", () => {
  assert.equal(normalise("  Catering &  Events. "), normalise("catering & events"));
});

test("finds a shared description and a description that repeats the title", async () => {
  const { pages, errors } = await collect(`${base}/sitemap-index.xml`, { concurrency: 2 });
  assert.equal(pages.length, 3);
  assert.equal(errors.length, 1);
  assert.match(errors[0].error, /^404/);
  const { repeats, sameAsTitle } = findRepeats(pages);
  assert.equal(repeats["og:description"].length, 1);
  assert.deepEqual(repeats["og:description"][0].urls.sort(), [`${base}/a`, `${base}/b`]);
  assert.equal(repeats["og:title"].length, 0);
  assert.deepEqual(sameAsTitle, [`${base}/c`]);
});

Running it with Node's built-in runner:

$ node --test share-dupes.test.mjs
✔ reads a sitemap index and a urlset (8.715901ms)
✔ takes the first og:title in the head, any attribute order (0.530901ms)
✔ normalise ignores case, spacing and trailing punctuation (5.729528ms)
✔ finds a shared description and a description that repeats the title (58.120496ms)
ℹ tests 4
ℹ suites 0
ℹ pass 4
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 269.339348

The last test is the one I care about most. Catering & events as the title and Catering & Events. as the description are the same line once decoded and normalised, and that is exactly the kind of near-copy a plain string comparison misses.

Reading the report

Run it against a real sitemap:

$ node share-dupes.mjs https://example.com/sitemap.xml

The output lists each repeated value once, with the URLs that use it, sorted so the biggest group comes first. That ordering matters. When forty URLs share one description, the cause is almost never forty editors making the same mistake. It is one template, one fallback, or one CMS field mapped to the wrong source. Fix that, and the whole group disappears at once.

Some patterns and what usually causes them:

• Every blog post has the site tagline as og:description. The theme falls back to the site description when the post excerpt is empty. Either write excerpts, or change the fallback to a trimmed first paragraph.
• Every product shares a title with its description. The feed maps the product name into both fields. Map the description from a short summary field instead, and leave it empty when that field is blank. An empty tag is easier to spot than a copy.
• Tag or category archives all share one title. The archive template prints the site name and nothing else. Include the term name.
• Paginated pages (/blog/page/2/) repeat page 1 exactly. That is often fine to leave alone, or to drop from the sitemap, since nobody shares page 7 of an archive.

Not every repeat is a bug. Legal pages, a contact page and a 404 template can reasonably share a generic description. The point of the report is to make those choices visible, not to force a unique sentence onto pages nobody will ever paste into a chat.

After you fix the template

Two checks before calling it done:

1. Run the script again and confirm the group is gone, or smaller for the right reasons.
2. Open two or three of the fixed URLs in a meta tag checker and read the title, meta description and Open Graph tags together. The script only compares strings. A person still has to read the new descriptions and decide whether they say something useful about the page.

Remember that messages already sent keep the card they were sent with. Fixing the template improves the next share, not the old ones.

What this script does not do

It does not render JavaScript. If your share tags are injected on the client, the script sees what most link preview crawlers see, which is the server HTML, and it will report the tags as missing. That is the correct result, even if it is not the one you hoped for.

It does not check images, sizes or redirects. Those are separate problems with separate tools, and keeping this script focused on text repeats is what keeps the report short enough to act on.

It does not do fuzzy matching, as covered above. If you want "these two descriptions are nearly the same", that is a reasonable extension, but start with the exact groups. On the sites I have run it on so far, the exact groups were the whole job.