python 使用BeautifulSoup的 html5lib爬取网站内容

1、使用BeautifulSoup的 'html5lib' 能像网页工具一样渲染内容。

缺点：运行比较慢

2、安装包

pip install html5lib

3、直接获取网页的所有有效内容

import requests #数据请求模块 第三方模块 pip install requests
from bs4 import BeautifulSoup
heads = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36'
}
def get_response(html_url):
    response = requests.get(url=html_url, headers=heads)
    return response

url= 'https://www.python100.com/html/1390.html'
response_new = get_response(html_url=url)
response_new.encoding = 'UTF-8'
#soup = BeautifulSoup(response_new.text, 'html.parser')
soup_new = BeautifulSoup(response_new.text, 'html5lib')
print(soup_new.text)

4、结果展示