爬取校园新闻首页的新闻的详情,使用正则表达式,函数抽离
1.爬取校园新闻首页新闻的标题、链接、正文、show-info。
标题:
print(soup.select('title'))
链接:
a=(soup.select('.news-list')) for b in a: c= b.a.attrs['href'] print(c)
正文:
d = soup.select('#content')[0].text print(d)
show-info:
info = soup.select('.show-info')[0].text print(info)
2.分析info字符串,获取每篇新闻的发布时间,作者,来源,摄影等信息
import requests from bs4 import BeautifulSoup url = 'http://news.gzcc.cn/html/xiaoyuanxinwen/' res = requests.get(url) res.encoding = 'utf-8' soup = BeautifulSoup(res.text, 'html.parser') for news in soup.select('li'): if len(news.select('.news-list-title')) > 0: title = news.select('.news-list-title')[0].text time = news.select('.news-list-info')[0].contents[0].text a = news.select('a')[0].attrs['href'] print(a,title,time) break res1 = requests.get(a) res1.encoding = 'utf-8' soup1 = BeautifulSoup(res1.text, 'html.parser') sp1 = soup1.select('#content')[0].text info = soup1.select('.show-info')[0].text print(info) dt = info.lstrip('发布时间:')[1:20] print(dt) ly = info.find('来源:') if ly>0: s = info[info.find('来源:'):].split()[0].lstrip('来源:') print(s) ly = info.find('摄影:') if ly>0: s = info[info.find('摄影:'):].split()[0].lstrip('摄影:') print(s)
使用正则表达式取得新闻编号:
import re newsurl='http://news.gzcc.cn/html/2018/xiaoyuanxinwen_0404/9183.html' print(re.match('http://news.gzcc.cn/html/2018/xiaoyuanxinwen_(.*).html',newsurl).group(0))
生成点击次数的Request URL:
def getNewsId(url): newsId = re.findall(r‘\_(.*).html‘, url)[0][-4:] clickUrl = ‘http://oa.gzcc.cn/api.php?op=count&id={}&modelid=80‘.format(newsId) clickRes = requests.get(clickUrl) # 利用正则表达式获取新闻点击次数 clickCount = int(re.search("hits‘\).html\(‘(.*)‘\);", clickRes.text).group(1)) return clickCount