DEV Community

Blogs World
Blogs World

Posted on

BeautifulSoup find() Returns None? Debug the HTML requests Got, Not the Page DevTools Shows

You copy a selector from Chrome DevTools, run it through BeautifulSoup, and get None (or an empty list from select()). The selector looks right. The element is right there in the Elements panel.

Almost every time, the problem is the same: DevTools shows the live DOM after the browser has run JavaScript, fixed broken markup and applied your headers and cookies. requests gives you the raw bytes the server sent to a Python script. Those are two different documents.

I rebuilt the four most common versions of this on a local test server and ran them with Python 3.13.5, requests 2.34.2, beautifulsoup4 4.15.0, lxml 6.1.3 and html5lib 1.1. Every output below is copied from those runs.

Step 0: look at what you actually received

Before you change a single selector, print three things:

import requests
from bs4 import BeautifulSoup

r = requests.get(url, timeout=10)
print(r.status_code, r.url)        # did you get a 200, and from the URL you expected?
print(len(r.text))
print(r.text[:500])                # what does the page really start with?

with open("got.html", "w", encoding="utf-8") as f:
    f.write(r.text)                # open this file and search it, not DevTools
Enter fullscreen mode Exit fullscreen mode

Search got.html for the text you want. Where it shows up (or doesn't) tells you which of the cases below you're in.

Case 1: the content is rendered by JavaScript

My test page has an empty <div id="app"> and a script that fills it in with <span class="price">$24.00</span>. In a browser, DevTools shows the span. In Python:

soup = BeautifulSoup(r.text, "html.parser")
print(soup.find("span", class_="price"))
print("$24.00" in r.text, '<span class="price"' in r.text)
Enter fullscreen mode Exit fullscreen mode
None
True True
Enter fullscreen mode Exit fullscreen mode

That second line is the trap. A plain "$24.00" in r.text check says the text is there, so the selector must be wrong, right? No. The text is there as a string inside a script, not as an element. Ask BeautifulSoup where the matches live:

import re
hits = soup.find_all(string=re.compile(r"\$24\.00"))
print([h.parent.name for h in hits])
print(soup.select_one("#app"))
Enter fullscreen mode Exit fullscreen mode
['script', 'script']
<div id="app"></div>
Enter fullscreen mode Exit fullscreen mode

If every match has script as its parent, no selector will ever work. You have two honest options:

  1. Read the data the page ships with. Many sites (Next.js, Nuxt, a lot of e-commerce themes) embed the data as JSON in a <script type="application/json"> or application/ld+json tag. Parse that instead of the HTML:
   import json
   data = json.loads(soup.find("script", id="__NEXT_DATA__").string)
   print(data["props"]["pageProps"]["products"][0])
Enter fullscreen mode Exit fullscreen mode
   {'name': 'Desk Lamp', 'price': '$24.00'}
Enter fullscreen mode Exit fullscreen mode

This is usually more stable than scraping markup, because class names change with every redesign and the JSON shape changes far less often. Also check the Network tab (filter by Fetch/XHR): the page might load the same data from a JSON endpoint you can call directly.

  1. Use a real browser (Playwright or Selenium) when the data only exists after scripts run. It's slower and heavier, so try option 1 first.

Case 2: tbody is in DevTools but not in the HTML

This one catches people who right-click → Copy selector on a table row. My test table has no <tbody> in the source, like a lot of hand-written and server-generated HTML:

<table id="t"><tr><td>Python</td><td>3.13</td></tr><tr><td>Node</td><td>22</td></tr></table>
Enter fullscreen mode Exit fullscreen mode

Browsers insert a <tbody> while parsing, so DevTools gives you table#t > tbody > tr. Here's that selector under each parser BeautifulSoup supports:

for parser in ["html.parser", "lxml", "html5lib"]:
    soup = BeautifulSoup(r.text, parser)
    print(parser, len(soup.select("table#t > tbody > tr")), len(soup.select("table#t tr")))
Enter fullscreen mode Exit fullscreen mode
html.parser 0 2
lxml 0 2
html5lib 2 2
Enter fullscreen mode Exit fullscreen mode

html.parser and lxml keep the source as written, so the copied selector finds nothing. html5lib follows the same HTML5 parsing rules as a browser and adds tbody, so it matches DevTools. The fix doesn't need a new parser, though: drop the strict > steps that came from DevTools and use table#t tr, which works with all three.

The general lesson: a selector copied from DevTools describes the browser's repaired tree. Trim it down to the parts that are really in got.html.

Case 3: multi-class elements and class_

Given <div class="price sale">$19.99</div>:

print(soup.find("div", class_="price"))
print(soup.find("div", class_="price sale"))
print(soup.find("div", class_="sale price"))
print(soup.select_one("div.sale.price"))
Enter fullscreen mode Exit fullscreen mode
<div class="price sale">$19.99</div>
<div class="price sale">$19.99</div>
None
<div class="price sale">$19.99</div>
Enter fullscreen mode Exit fullscreen mode

A single class name matches any element that has that class. A string with a space in it is compared to the whole class attribute, in that exact order, so "sale price" misses. If you need several classes, use a CSS selector (div.sale.price), where order doesn't matter. And it's class_, with the underscore: class is a reserved word in Python.

Case 4: the server sent you a different page

My /guarded route acts like many real sites: it returns a "Please enable JavaScript" page with a 403 when it sees the default requests User-Agent.

r = requests.get(url, timeout=10)
print(r.status_code, r.request.headers["User-Agent"])
print(BeautifulSoup(r.text, "html.parser").find("td"))
Enter fullscreen mode Exit fullscreen mode
403 python-requests/2.34.2
None
Enter fullscreen mode Exit fullscreen mode

Your selector is fine. You're parsing an error page. Two habits make this obvious instead of mysterious:

r.raise_for_status()
Enter fullscreen mode Exit fullscreen mode
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://127.0.0.1:18923/guarded
Enter fullscreen mode Exit fullscreen mode

Call it right after every request, so a 403, 404 or 500 fails loudly at the request instead of quietly as None three functions later. And check r.url too: a 200 from a login or cookie-consent page you got redirected to is just as wrong.

On my test server, sending a normal browser User-Agent returned the real page (200, and find("td") gave <td>Python</td>). On real sites, treat a block as information: read the site's terms and robots.txt, use an official API if there is one, and keep your request rate low. Don't treat "send more headers" as a way to get around a site that has said no.

The checklist

When find() returns None:

  1. raise_for_status() and print r.url. Are you even on the right page?
  2. Save r.text to a file and search it. Is the text there at all?
  3. If it's only inside script tags, look for embedded JSON or the XHR endpoint before reaching for a headless browser.
  4. Remove tbody and long > chains copied from DevTools.
  5. For several classes, use select("tag.a.b") instead of class_="a b".
  6. Guard the result before you call .text on it, so a missing element gives you a clear message instead of AttributeError: 'NoneType' object has no attribute 'text':
el = soup.select_one("div.sale.price")
if el is None:
    raise ValueError("price not found. Check got.html: layout changed or the page is JS-rendered")
price = el.get_text(strip=True)
Enter fullscreen mode Exit fullscreen mode

If you're new to scraping, I wrote a longer walkthrough on Blogs World covering setup, CSS selectors, pagination and saving to CSV: How to Scrape a Website with Python and BeautifulSoup. That NoneType error from step 6 is also covered in Common Python Errors and How to Fix Them. When the data turns out to be embedded JSON, the free JSON formatter makes it much easier to find the path you need before you write data["props"][...].

Top comments (0)