股票数据Scrapy爬虫实例（亲测有效）

步骤：

步骤1：建立工程和Spider模板

• scrapy startproject BaiduStocks
• cd BaiduStocks
• scrapy genspider stocks baidu.com
• 进一步修改spiders/stocks.py

这一步自行完成~

步骤2：编写Spider

• 配置stocks.py文件
• 修改对返回页面的处理
• 修改对新增URL爬取请求的处理（stocks.py）

# -*- coding: utf-8 -*-
import scrapy
import re


class StocksSpider(scrapy.Spider):
    name = 'stocks'
    start_urls = ['http://quote.eastmoney.com/stock_list.html']

    def parse(self, response):
        for href in response.css('a::attr(href)').extract():
            try:
                stock = re.findall(r"[s][hz]\d{6}", href)[0]
                url = 'http://gu.qq.com/' + stock + '/gp'
                yield scrapy.Request(url, callback=self.parse_stock)
            except:
                continue

    def parse_stock(self, response):
        infoDict = {}
        stockName = response.css('.title_bg')
        stockInfo = response.css('.col-2.fr')
        name = stockName.css('.col-1-1').extract()[0]
        code = stockName.css('.col-1-2').extract()[0]
        info = stockInfo.css('li').extract()
        for i in info[:13]:
            key = re.findall('>.*?<', i)[1][1:-1]
            key = key.replace('\u2003', '')
            key = key.replace('\xa0', '')
            try:
                val = re.findall('>.*?<', i)[3][1:-1]
            except:
                val = '--'
            infoDict[key] = val

        infoDict.update({'股票名称': re.findall('\>.*\<', name)[0][1:-1] + \
                                 re.findall('\>.*\<', code)[0][1:-1]})
        yield infoDict

其中的key=re.replace('\u2003','')，key=re.replace('\xa0','')分别是为了除去爬取的字符串中的无用部分，如&nbsp等，网页抓取时会因为编码原因转化成\xa0，所以我们需要进行替换，得到较为美观的字符串.

步骤3：编写ITEM Pipelines

• 配置pipelines.py文件
• 定义对爬取项（Scrapy Item）的处理类（pipelines.py）

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


class ScrapyGupiaoPipeline:
    def process_item(self, item, spider):
        return item

class ScrapyGupiaoPipeline:
    def open_spider(self, spider):
        self.f = open('gupiao.txt', 'w')

    def close_spider(self, spider):
        self.f.close()

    def process_item(self, item, spider):
        try:
            line = str(dict(item)) + '\n'
            self.f.write(line)
        except:
            pass
        return item

步骤四：配置ITEM_PIPELINES选项（settings.py）

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'BaiduStocks.pipelines.ScrapyGupiaoPipeline': 300,
}