How to check every URL in your sitemap.xml for 404s, redirect chains and timeouts (a Python script for CI)

A sitemap is a list of URLs you're telling the world are live, canonical and worth visiting. In practice, sitemaps drift. A page gets deleted but the generator still lists it from a stale cache. A slug changes and the sitemap keeps the old one, which now redirects. A section moves behind a login. Nobody notices, because nobody reads sitemaps except crawlers.

Checking a sitemap is also one of the easiest link checks you can automate. You don't need to crawl anything: the list of URLs is already there. This post builds a small Python script that reads a sitemap (including sitemap indexes), fetches every URL with sensible limits, and reports four things:

• URLs that return an error (4xx, 5xx).
• URLs that redirect at all (a sitemap should list final URLs).
• Redirect chains of more than one hop.
• URLs that time out or fail to connect.

It exits non-zero when it finds problems, so you can run it in CI after a deploy.

Reading the sitemap

Sitemaps come in two shapes: a <urlset> with <url><loc> entries, or a <sitemapindex> that points to more sitemaps. Large sites nearly always use an index. The XML namespace trips people up, so handle it explicitly:

import xml.etree.ElementTree as ET
import gzip, urllib.request

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def fetch_bytes(url, timeout=20):
    req = urllib.request.Request(url, headers={"User-Agent": "sitemap-check/1.0"})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        data = r.read()
    return gzip.decompress(data) if url.endswith(".gz") else data

def sitemap_urls(url, seen=None):
    seen = seen or set()
    if url in seen:
        return []
    seen.add(url)
    root = ET.fromstring(fetch_bytes(url))
    if root.tag.endswith("sitemapindex"):
        out = []
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            out += sitemap_urls(loc.text.strip(), seen)
        return out
    return [loc.text.strip() for loc in root.findall("sm:url/sm:loc", NS)]

The seen set protects you from an index that accidentally lists itself, which happens more often than you'd think with plugins.

Checking each URL without following redirects blindly

Most HTTP libraries follow redirects silently. For a sitemap check, you want to see every hop, so follow them yourself. This uses requests because it makes that easy:

import requests

def check(url, max_hops=5, timeout=10):
    hops, current = [], url
    s = requests.Session()
    s.headers["User-Agent"] = "sitemap-check/1.0 (+your contact URL)"
    for _ in range(max_hops + 1):
        try:
            r = s.get(current, allow_redirects=False, timeout=timeout, stream=True)
            r.close()
        except requests.Timeout:
            return {"url": url, "status": "timeout", "hops": hops}
        except requests.ConnectionError as e:
            return {"url": url, "status": "connect-error", "hops": hops, "detail": str(e)[:120]}
        if r.is_redirect or r.status_code in (301, 302, 303, 307, 308):
            nxt = requests.compat.urljoin(current, r.headers.get("Location", ""))
            hops.append((r.status_code, nxt))
            current = nxt
            continue
        return {"url": url, "status": r.status_code, "hops": hops, "final": current}
    return {"url": url, "status": "too-many-redirects", "hops": hops}

Two details matter here. stream=True plus close() means you don't download a 40 MB PDF just to learn it exists. And using GET rather than HEAD avoids the servers that answer HEAD differently, or not at all.

Running it in parallel, politely

A few thousand URLs one at a time takes a while. A small thread pool is enough, and keeping it small stops your own check from looking like an attack on your own site:

from concurrent.futures import ThreadPoolExecutor
import sys, time

def main(sitemap, workers=8):
    urls = sorted(set(sitemap_urls(sitemap)))
    print(f"{len(urls)} URLs in sitemap")
    t0 = time.time()
    with ThreadPoolExecutor(max_workers=workers) as pool:
        results = list(pool.map(check, urls))
    problems = 0
    for r in results:
        st, hops = r["status"], r["hops"]
        if isinstance(st, int) and st >= 400:
            print(f"ERROR {st}  {r['url']}"); problems += 1
        elif not isinstance(st, int):
            print(f"FAIL  {st}  {r['url']}"); problems += 1
        elif len(hops) > 1:
            print(f"CHAIN {len(hops)} hops  {r['url']} -> {r['final']}"); problems += 1
        elif hops:
            print(f"REDIR {hops[0][0]}  {r['url']} -> {r['final']}"); problems += 1
    print(f"checked {len(urls)} in {time.time()-t0:.1f}s, {problems} problems")
    return 1 if problems else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Run it with python sitemap_check.py https://example.com/sitemap.xml.

What the results usually show

On real sites, the first run tends to surface the same categories:

• Single redirects. By far the most common. http:// URLs in an otherwise HTTPS sitemap, missing or extra trailing slashes, and old slugs. These aren't broken for users, but a sitemap that lists redirecting URLs is telling crawlers the wrong canonical. Fix the generator, not the redirect.
• Chains. Usually http://example.com/old to https://example.com/old to https://example.com/new/. Each migration added a hop. Point the first rule straight at the final URL.
• 404s for deleted content. The generator is reading from a cache, a search index or a table that doesn't respect the "deleted" flag.
• Timeouts on a handful of heavy pages. Often the slowest pages on the site, and worth knowing about regardless of the sitemap.

Putting it in CI

Run it after deploy against production or a preview environment, not before, because you're testing what's actually served:

sitemap-check:
  runs-on: ubuntu-latest
  needs: deploy
  steps:
    - uses: actions/checkout@v4
    - run: pip install requests
    - run: python scripts/sitemap_check.py https://example.com/sitemap.xml

If your sitemap is large, check a random sample on every deploy and the whole thing nightly. If single redirects are noisy at first, treat them as warnings for a week while you fix the generator, then make them fail the build.

Limits of this approach

A sitemap check only sees URLs you've listed. It won't find broken links inside your pages, links to pages missing from the sitemap, or broken outbound links. It's one useful slice. For the rest you still need a crawl or a dedicated tool; checking the links inside pages, and the external ones, is the part linkbruno.com focuses on.

Related reading

• Redirect chains: why 3 hops is 2 too many (and a script that finds them)
• HEAD vs GET for link checking: HEAD is faster, and it lies more than you'd think
• How to find broken links with Playwright (a test you can drop into CI)
• Should you fix dead links in old blog posts, or leave them as a historical record? Let's argue about it

Have you ever run a check like this against your own sitemap? What share of the URLs redirected on the first run?