DEV Community

Siftwright
Siftwright

Posted on

Why your email scraper misses addresses: Cloudflare [email protected], entities and [at]/[dot] in Python

#python #webscraping #cloudflare #tutorial

You scrape a company's contact page, run an email regex over it, and get nothing back. Then you open the page in a browser and there is the address, right in front of you.

Usually the address is there but hidden from naive scrapers. The four cases I see most:

  1. Cloudflare Email Address Obfuscation: the HTML says [email protected], and the real address is hex-encoded in a data-cfemail attribute or a /cdn-cgi/l/email-protection#... link.
  2. HTML entities: support@example.com renders as support@example.com, but a regex over the raw source never sees the @.
  3. Hand-written "at/dot": jobs [at] example [dot] co [dot] uk.
  4. Structured data: the address only appears in a JSON-LD contactPoint, never in visible text.

Below is one dependency-free Python script that handles all four, plus the one case it deliberately doesn't.

How Cloudflare's obfuscation works

When a site behind Cloudflare has Email Address Obfuscation switched on, Cloudflare rewrites every address in the HTML to something like:

<span class="__cf_email__" data-cfemail="5a2a283f29291a3f223b372a363f74393537">[email&#160;protected]</span>
Enter fullscreen mode Exit fullscreen mode

A small script on the page decodes it in the browser. The encoding is a single-byte XOR: the first byte is the key, and every following byte XOR the key is one character of the address. So decoding is three lines:

def decode_cfemail(hexstr: str) -> str:
    data = bytes.fromhex(hexstr)
    key = data[0]
    return bytes(b ^ key for b in data[1:]).decode("utf-8", "replace")

decode_cfemail("5a2a283f29291a3f223b372a363f74393537")  # 'press@example.com'
Enter fullscreen mode Exit fullscreen mode

The same hex appears in links as href="/cdn-cgi/l/email-protection#<hex>", so match both forms.

The full script 

import html, json, re, sys, urllib.request

EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
CF_ATTR = re.compile(r'data-cfemail="([0-9a-fA-F]+)"')
CF_HREF = re.compile(r'/cdn-cgi/l/email-protection#([0-9a-fA-F]+)')
AT_DOT = re.compile(
    r"\b([A-Za-z0-9._%+-]+)\s*[\[\(\{]\s*at\s*[\]\)\}]\s*"
    r"([A-Za-z0-9-]+(?:\s*[\[\(\{]\s*dot\s*[\]\)\}]\s*[A-Za-z0-9-]+)+)",
    re.I,
)
DOT = re.compile(r"\s*[\[\(\{]\s*dot\s*[\]\)\}]\s*", re.I)
JSONLD = re.compile(r'<script[^>]+application/ld\+json[^>]*>(.*?)</script>', re.S | re.I)
JUNK_TLDS = ("png", "jpg", "jpeg", "gif", "svg", "webp", "css", "js")


def decode_cfemail(hexstr: str) -> str:
    """Cloudflare Email Address Obfuscation: byte 0 is an XOR key for the rest."""
    data = bytes.fromhex(hexstr)
    key = data[0]
    return bytes(b ^ key for b in data[1:]).decode("utf-8", "replace")


def walk_jsonld(node, out):
    if isinstance(node, dict):
        for k, v in node.items():
            if k.lower() == "email" and isinstance(v, str):
                out.add(v.removeprefix("mailto:").strip())
            else:
                walk_jsonld(v, out)
    elif isinstance(node, list):
        for v in node:
            walk_jsonld(v, out)


def find_emails(raw_html: str) -> dict:
    found = {}

    def add(addr, how):
        addr = addr.strip().strip(".").lower()
        if EMAIL.fullmatch(addr) and not addr.rsplit(".", 1)[-1] in JUNK_TLDS:
            found.setdefault(addr, how)

    # 1. Cloudflare-obfuscated addresses (span attribute and link form)
    for h in CF_ATTR.findall(raw_html) + CF_HREF.findall(raw_html):
        try:
            add(decode_cfemail(h), "cloudflare")
        except ValueError:
            pass
    # 2. JSON-LD structured data
    for block in JSONLD.findall(raw_html):
        try:
            emails = set()
            walk_jsonld(json.loads(block), emails)
            for e in emails:
                add(e, "json-ld")
        except json.JSONDecodeError:
            pass
    # 3. Entity-encoded text and mailto: links (html.unescape turns &#64; into @)
    text = html.unescape(raw_html)
    for m in re.findall(r'mailto:([^"\'?>\s]+)', text, re.I):
        add(urllib.request.unquote(m), "mailto")
    for m in EMAIL.findall(text):
        add(m, "text")
    # 4. "name [at] example [dot] com" written out by hand
    plain = re.sub(r"<[^>]+>", " ", text)
    for user, domain in AT_DOT.findall(plain):
        add(f"{user}@{DOT.sub('.', domain)}", "at-dot")
    return found


if __name__ == "__main__":
    url = sys.argv[1]
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (contact-finder example)"})
    page = urllib.request.urlopen(req, timeout=20).read().decode("utf-8", "replace")
    for addr, how in find_emails(page).items():
        print(f"{addr:40} {how}")
Enter fullscreen mode Exit fullscreen mode

A few details that save you from false positives:

  • html.unescape before the regex turns &#64; and &commat; into @, so entity-encoded addresses match.
  • Image names like logo@2x.png look like emails. The JUNK_TLDS check drops them.
  • mailto: values are URL-encoded (hello%40example.net?subject=Hi), so unquote them and cut at ?.
  • The "at/dot" pattern requires brackets ([at], (at), {at}). Matching a bare at would turn ordinary sentences into fake addresses.

Testing it

I couldn't find a public page that uses Cloudflare obfuscation and that I'd be comfortable naming, so the test below builds one. It encodes addresses exactly as Cloudflare does and mixes in the other cases:

from find_emails import find_emails

def encode_cfemail(addr, key=0x5A):
    return f"{key:02x}" + "".join(f"{ord(c) ^ key:02x}" for c in addr)

page = f'''
<p>Press: <span class="__cf_email__" data-cfemail="{encode_cfemail('press@example.com')}">[email&#160;protected]</span></p>
<a href="/cdn-cgi/l/email-protection#{encode_cfemail('sales@example.org', 0x21)}">Sales</a>
<p>Support: support&#64;example.com</p>
<p>Jobs: jobs [at] example [dot] co [dot] uk</p>
<a href="mailto:hello%40example.net?subject=Hi">Say hi</a>
<img src="logo@2x.png">
<script type="application/ld+json">{{"@type":"Organization","contactPoint":{{"email":"mailto:info@example.com"}}}}</script>
'''
for addr, how in find_emails(page).items():
    print(f"{addr:32} {how}")
Enter fullscreen mode Exit fullscreen mode

Output (real run):

press@example.com                cloudflare
sales@example.org                cloudflare
info@example.com                 json-ld
hello@example.net                mailto
support@example.com              text
jobs@example.co.uk               at-dot
Enter fullscreen mode Exit fullscreen mode

Six addresses, no logo@2x.png. On a live page it works the same way, e.g. python find_emails.py https://www.python.org/about/help/ prints the mailto: addresses published there.

What this deliberately doesn't do

  • JavaScript-assembled addresses ('user' + '@' + 'example.com' inside a script) need a headless browser. Evaluating arbitrary page scripts to dig out an address is rarely worth it.
  • Contact forms and images. If the only way to reach a company is a form or a PNG of an address, that's the company's choice. Use the form.
  • Guessing. It never invents first.last@company.com patterns and doesn't check whether a mailbox exists.

Finding a published address isn't permission to send marketing email. GDPR/ePrivacy in Europe and CAN-SPAM in the US still apply, so keep messages relevant and honour opt-outs.

When you don't want to maintain this yourself

  • For a one-off lookup, the free website contact finder checks a site's homepage and contact/about pages for emails, phones and social links, with no signup.
  • For a list of companies, the Contact Details Extractor on Apify takes a list of domains and returns JSON or CSV at $2 per 1,000 websites. Sites with nothing to return aren't billed.

Both are ours (Siftwright). The script above is free to copy, so use whichever fits.

Top comments (0)