python 使用BeautifulSoup的 html5lib爬取网站内容

1、使用BeautifulSoup的 'html5lib' 能像网页工具一样渲染内容。

缺点：运行比较慢

2、安装包

?

1	pip install html5lib

3、直接获取网页的所有有效内容

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

import requests #数据请求模块 第三方模块 pip install requests from bs4 import BeautifulSoup heads = {     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36' } def get_response(html_url):     response = requests.get(url=html_url, headers=heads)     return response   url= 'https://www.python100.com/html/1390.html' response_new = get_response(html_url=url) response_new.encoding = 'UTF-8' #soup = BeautifulSoup(response_new.text, 'html.parser') soup_new = BeautifulSoup(response_new.text, 'html5lib') print(soup_new.text)

4、结果展示