DEV Community

drake
drake

Posted on

非标准HTML无法被解析的问题解决

当爬虫请求一个网页，这个网页是非标准HTML的时候，那么一般方式都是无法正常解析成dom的；比如：

错误示例

  • 1、此种方式将得到None 
from lxml import etree
tree = etree.HTML(res.text)
Enter fullscreen mode Exit fullscreen mode
  • 2、此种方式也是得到None 
from lxml import etree

parser = etree.HTMLParser()
tree = etree.fromstring(res.text, parser)
Enter fullscreen mode Exit fullscreen mode
  • 3、此种方式也是得到None 
from bs4 import BeautifulSoup
tree = BeautifulSoup(html, 'html.parser')
Enter fullscreen mode Exit fullscreen mode

正确示例

  • 1、解决方案 
from lxml.html import soupparser
tree = soupparser.fromstring(res.text)
Enter fullscreen mode Exit fullscreen mode
  • 2、解决方案 
from lxml import etree
tree = etree.HTML(res.text.encode("ascii", "xmlcharrefreplace").decode("ascii"))
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

harman_diaz profile image

Top Cloud Readiness Assessment Challenges and How to Overcome Them

Harman Diaz -

mrcaption49 profile image

Constraints in Oracle SQL

Pranav Bakare -

sonu_sindhi profile image

i have devloped portfolio website sonusindhi.netlify.app. request all of you to review all page and suggest for improvement. Am learning, will continue learn.

Sonu Sindhi -

cdfortes profile image

Site Initializers: Automação e Personalização do Ambiente de Desenvolvimento

Carlos Fortes -