2019基于python的网络爬虫系列，爬取糗事百科

**因为糗事百科的URL改变，正则表达式也发生了改变，导致了网上许多的代码不能使用，所以写下了这一篇博客，希望对大家有所帮助，谢谢！**

废话不多说，直接上代码。

为了方便提取数据，我用的是beautifulsoup库和requests


``## 具体代码如下

```
import requests
from bs4 import BeautifulSoup

def download_page(url):
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0"}
r = requests.get(url, headers=headers)
return r.text

def get_content(html):
soup = BeautifulSoup(html, 'html.parser')
con = soup.find(id='main')
con_list = con.find_all('div', class_="cat_llb")
for i in con_list:
author = i.find('h3').string # 获取名字
content = i.find('div', id="endtext").get_text() # 获取内容
save_txt(author, content)

def save_txt(*args):
for i in args:
with open('qiubai.txt', 'a', encoding='utf-8') as f:

f.write(i+'\n'+'\n')

# def save_txt(str):
# for i in str:
#
# with open('qiubai.txt', 'a', encoding='utf-8') as f:
# f.write(str + '\n')
# f.write(i)

 

def main():
# 可以构造如下 url，

for i in range(1, 20):

url = 'http://www.lovehhy.net/Joke/Detail/QSBK/{}'.format(i)
html = download_page(url)
get_content(html)

if __name__ == '__main__':
main()

```

哦 ,对了，新网站的地址是http://www.lovehhy.net/Joke/Detail/QSBK/
有什么不懂得欢迎留言