Python爬虫小实例：爬股票数据

在上一篇博客中，我们介绍了爬高校排名的爬虫程序，本篇博客我们将介绍爬股票数据的程序。

程序来源：中国大学MOOC网《网络爬虫与信息提取课程》。

程序目的：获取上交所和深交所的部分股票信息，输出到文件。

读懂以下程序需提前了解requests库、BeautifulSoup库和re库，在《网络爬虫与信息提取课程》有提供相关知识。

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67

import requests from bs4 import BeautifulSoup import re     def getHTMLText(url, code="utf-8"):     try:         r = requests.get(url)         r.raise_for_status()         r.encoding = code     # 直接指定用utf-8解码         return r.text     except:         return ""     def getStockList(lst, stockURL):     html = getHTMLText(stockURL, "GB2312")    # 东方财富网用的是GB2312编码     soup = BeautifulSoup(html, 'html.parser')     a = soup.find_all('a')[:200]    # 股票代码在a标签中的href属性中。我们只取前100个a标签     for i in a:         try:             href = i.attrs['href']             lst.append(re.findall(r"\d{6}", href)[0])         except:             continue     def getStockInfo(lst, stockURL, fpath):     count = 0     for stock in lst:         url = stockURL + stock    # 每只股票的信息的链接         html = getHTMLText(url)         try:             if html == "":                 continue             infoDict = {}    # 每只股票的信息都存在字典中             soup = BeautifulSoup(html, 'html.parser')             stockInfo = soup.find('div', attrs={'class': 'stock-info'})             name = stockInfo.find_all(attrs={'class': 'stock-name'})[0]             infoDict.update({'股票名称': name.text.split()[0]})   # .text可以取出标签中的字符串             keyList = stockInfo.find_all('dt')[:4]     # 只取股票的前4个信息             valueList = stockInfo.find_all('dd')[:4]             for i in range(len(keyList)):                 key = keyList[i].text                 val = valueList[i].text                 infoDict[key] = val             with open(fpath, 'a', encoding='utf-8') as f:                 f.write(str(infoDict) + '\n')                 count = count + 1                 print("\r当前进度: {:.2f}%".format(count * 100 / len(lst)), end="")   # \r表示返回到当前行的起始位置，打印的内容会覆盖之前打印的内容                                                                                      # 每次print之后，会自动换行。end用来取消自动换行。         except:             count = count + 1             print("\r当前进度: {:.2f}%".format(count * 100 / len(lst)), end="")             continue     def main():     stock_list_url = 'https://quote.eastmoney.com/stock_list.html'   # 从东方财富网获取每只股票的代码     stock_info_url = 'https://www.laohu8.com/stock/'                 # 从老虎社区网站获取每只股票的信息     output_file = '/Users/wangpeng/Desktop/BaiduStockInfo.txt'     slist = []     getStockList(slist, stock_list_url)     getStockInfo(slist, stock_info_url, output_file)     main()

文件结果：