初识scrapy框架（三）--------crawlspider

　　为了说明crawlspider 与 spider 的不同，我们以初识scrapy框架（二）的案例来跟进代码的书写。

创建爬虫文件：scrapy genspider -t crawl 'crawlspider_name'  'url'  .

编写爬虫文件：

from Tencent_recruit.items import TencentRecruitItem, DetailRecruitItem
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class CrawlRectSpider(CrawlSpider):
    name = 'crawl_rect'
    allowed_domains = ['hr.tencent.com']
    start_urls = ['https://hr.tencent.com/position.php']

    rules = (
        Rule(LinkExtractor(allow=r'position\.php\?&start=\d+#a'), callback='parse_tencent', follow=True),
        Rule(LinkExtractor(allow=r'position_detail\.php\?id=\d+&keywords=&tid=0&lid=0'), callback='parse_detail', follow=False),
    )

    def parse_tencent(self, response):
        node_list = response.xpath('//tr[@class="even"] | //tr[@class="odd"]')
        for node in node_list:
            item = TencentRecruitItem()
            item['position_name'] = node.xpath('./td[1]/a/text()').extract_first()
            item['position_category'] = node.xpath('./td[2]/text()').extract_first()
            item['position_number'] = node.xpath('./td[3]/text()').extract_first()
            item['position_place'] = node.xpath('./td[4]/text()').extract_first()
            item['position_link'] = node.xpath('./td[1]/a/@href').extract_first()
            item['release_time'] = node.xpath('./td[5]/text()').extract_first()
            yield item

    def parse_detail(self, response):
        item = DetailRecruitItem()
        print('detail正在执行')
        item['position_name'] = response.xpath('//*[@id="sharetitle"]/text()').extract_first()
        result = response.xpath('//ul[@class="squareli"]')
        duty = result[0]
        req = result[1]
        item['work_duty'] = ''.join(duty.xpath('./li/text()').extract())
        item['work_request'] = ''.join(req.xpath('./li/text()').extract())
        yield item

执行爬虫程序：scrapy crawl crawl_rect

　　执行过程可以看出，比spider更快，代码两者相比较crawlspider更少，更方便！！

crawlspider 与 spider 的总结：