DEV Community

drake
drake

Posted on • Edited on

非标准HTML无法被解析的问题解决

当爬虫请求一个网页，这个网页是非标准HTML的时候，那么一般方式都是无法正常解析成dom的；比如：

错误示例

  • 1、此种方式将得到None 
from lxml import etree
tree = etree.HTML(res.text)
Enter fullscreen mode Exit fullscreen mode
  • 2、此种方式也是得到None 
from lxml import etree

parser = etree.HTMLParser()
tree = etree.fromstring(res.text, parser)
Enter fullscreen mode Exit fullscreen mode
  • 3、此种方式也是得到None 
from bs4 import BeautifulSoup
tree = BeautifulSoup(html, 'html.parser')
Enter fullscreen mode Exit fullscreen mode

正确示例

  • 1、解决方案

原理是修复缺损的HTML，以及修复非标准的HTML

from lxml.html import soupparser
tree = soupparser.fromstring(res.text)
Enter fullscreen mode Exit fullscreen mode
  • 2、解决方案

原理是HTML文本中存在非标准ASCCI码导致解析异常，将其统一转换成ASCCI码

from lxml import etree
tree = etree.HTML(res.text.encode("ascii", "xmlcharrefreplace").decode("ascii"))
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

jguerrero-voxel51 profile image

ECCV 2024 - Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models

Jimmy Guerrero -

ayoola_damilare_212d5bde0 profile image

My React Journey: Day 5

Ayoola Damilare -

swathi_nit_c44b7f9b5d121 profile image

N queen problem - Backtracking

SWATHI N -

keerthana_r_28564ebdaf441 profile image

🚗 Solving Traffic Congestion with Hamiltonian Circuits

Keerthana R -