Scrapy框架CrawlSpider类爬虫实例

CrawlSpider类爬虫中：

rules用于定义提取URl地址规则，元祖数据有顺序

    #LinkExtractor 连接提取器，提取url地址

　 #callback 提取出来的url地址的response会交给callback处理

　#follow 当前url地址的响应是否重新经过rules进行提取url地址

cf.py具体实现代码如下（简化版）：

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
import re

class CfSpider(CrawlSpider):
    name = 'cf'
    allowed_domains = ['bxjg.circ.gov.cn']
    start_urls = ['http://bxjg.circ.gov.cn/web/site0/tab5240/Default.htm']

    rules = (
        Rule(LinkExtractor(allow=r'/web/site0/tab5240/info\d+\.htm'), callback='parse_item', ),
        Rule(LinkExtractor(allow=r'/web/site0/tab5240/module14430/page\d+\.htm'),follow=True, ),
    )

    def parse_item(self, response):
        item = {}
        item['title'] = re.findall("<!--TitleStart-->(.*?)<!--TitleEnd-->", response.body.decode())[0]
        item['publish_date'] = re.findall("发布时间：(20\d{2}-\d{2}-\d{2})", response.body.decode())[0]
        print(item)