Scrapy抓取起点中文网排行榜
项目名称：qidian
项目描述：利用scrapy抓取七点中文网的“完本榜”总榜的500本小说，抓取内容包括：小说名称，作者，类别，然后保存为CSV文件
目标URL：https://www.qidian.com/rank/fin?style=1
项目需求：
1.小说名称
2.作者
3.小说类别
第一步：在shell中创建项目
scrapy startproject qidian
第二步：根据项目需求编辑items.py
1 #-*- coding: utf-8 -*- 2 import scrapy 3 4 class QidianItem(scrapy.Item): 5 name = scrapy.Field() 6 author = scrapy.Field() 7 category = scrapy.Field()
第三步：进行页面分析，利用xpath或者css提取数据，创建并编辑spider.py
1 # -*- coding: utf-8 -*- 2 import scrapy 3 from ..items import QidianItem 4 5 class QidianSpider(scrapy.Spider): 6 name = 'qidian' 7 start_urls = ['https://www.qidian.com/rank/fin?style=1&dateType=3'] 8 9 def parse(self, response): 10 sel = response.xpath('//div[@class="book-mid-info"]') 11 for i in sel: 12 name = i.xpath('./h4/a/text()').extract_first() 13 author = i.xpath('./p[@class="author"]/a[1]/text()').extract_first() 14 category = i.xpath('./p[@class="author"]/a[last()]/text()').extract_first() 15 item = QidianItem() 16 item['name'] = name 17 item['author'] = author 18 item['category'] = category 19 yield item
上面这里是一页的数据，接下来抓取一下页的连接（因为项目过于小巧，我认为没必要用到一些高大上的方法来实现，直接观察URL的构造规律就可以简单写出代码），下面是spider.py的完整代码
1 # -*- coding: utf-8 -*- 2 import scrapy 3 from ..items import QidianItem 4 5 class QidianSpider(scrapy.Spider): 6 name = 'qidian' 7 start_urls = ['https://www.qidian.com/rank/fin?style=1&dateType=3'] 8 n = 1 #第一页 9 10 def parse(self, response): 11 sel = response.xpath('//div[@class="book-mid-info"]') 12 for i in sel: 13 name = i.xpath('./h4/a/text()').extract_first() 14 author = i.xpath('./p[@class="author"]/a[1]/text()').extract_first() 15 category = i.xpath('./p[@class="author"]/a[last()]/text()').extract_first() 16 item = QidianItem() 17 item['name'] = name 18 item['author'] = author 19 item['category'] = category 20 yield item
21 22 if self.n < 25: 23 self.n += 1 #n表示页码 24 next_url = 'https://www.qidian.com/rank/fin?style=1&dateType=3&page=%d' % self.n 25 yield scrapy.Request(next_url, callback = parse)
第四步：启动爬虫并保存数据
scrapy crawl qidian -o qidian.csv
