利用Scrapy爬取自己的CSDN博客

      最近开始接触Scrapy这个开源的爬虫，看了一些文档和人家的技术博客，模仿一下，来爬取自己博客。

      首先创建项目：

scrapy startproject myblog

      items.py的编写：

     我准备爬取博客文章标题，文章链接及文章被阅读的次数

# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html
import scrapy
class MyBlogItem(scrapy.Item):
    article_name = scrapy.Field()
    article_url = scrapy.Field()
    article_readcount = scrapy.Field()

      pipelines.py的编写：

# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
import json
import codecs
class MyBlogPipeline(object):
    def __init__(self):
        self.file = codecs.open('myblog_data.json',mode='wb',encoding='utf-8')
    def process_item(self, item, spider):
        line = json.dumps(dict(item))+'\n'
        self.file.write(line.decode('unicode_escape'))
        return item

      Scrapy爬虫框架抓取的中文结果为Unicode编码，对于如何转换为UTF-8编码。下面部分的代码算是比较好的解决了这个问题。

      settings.py的编写：

# -*- coding: utf-8 -*-
# Scrapy settings for myblog project
#
# For simplicity, this file contains only the most important settings by
# default. All the other settings are documented here:
#
#     http://doc.scrapy.org/en/latest/topics/settings.html
#
BOT_NAME = 'myblog'
SPIDER_MODULES = ['myblog.spiders']
NEWSPIDER_MODULE = 'myblog.spiders'
COOKIES_ENABLED = False
ITEM_PIPELINES = {
    'myblog.pipelines.MyBlogPipeline':300
}
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'myblog (+http://www.yourdomain.com)'

      这里将COOKIES_ENABLED参数置为True，使根据cookies判断访问的站点不能发现爬虫轨迹，防止被ban。

      ITEM_PIPELINES类型为字典，用于设置启动的pipeline，其中key为定义的pipeline类，value为启动顺序，默认0-1000。

      爬虫的编写：

#!/usr/bin/env python
# __author__ = 'root'
from scrapy.spider import Spider
from scrapy.selector import  Selector
from scrapy.http import Request
from myblog.items import MyBlogItem
import  re
class MyBlogSpider(Spider):
    name = "myblog"
    download_delay = 1
    allowed_domains=["blog.csdn.net"]
    start_urls=[
        "http://blog.csdn.net/bnxf00000/article/details/2785136"
    ]
    def parse(self, response):
        sel = Selector(response)
        item = MyBlogItem()
        templist=[]
        article_url = str(response.url)
        article_name = sel.xpath('//div[@id="article_details"]/div/h1/span/a/text()').extract()
        article_readcount = sel.xpath('//div[@id="article_details"]/div[2]/span[@class="link_view"]/text()').extract()
        for temp in article_readcount:
            result = re.match('(\d+)',temp)
            if result:
                templist.append(result.group(0))
        #article_readcount = re.match('\d+',article_readcount)
        item['article_name'] = [n.encode('utf-8') for n in article_name]
        item['article_url'] = article_url.encode('utf-8')
        item['article_readcount']=[n.encode('utf-8') for n in templist]
        yield item
        urls = sel.xpath('//li[@class="next_article"]/a/@href').extract()
        for url in urls:
            #print url
            url = "http://blog.csdn.net" + url
            #print url
            yield Request(url, callback=self.parse)

      原理是分析网页得到“下一篇”的链接，并返回Request对象。进而继续爬取下一篇文章，直至没有。

      执行：

scrapy crawl myblog

      部分结果图示：

      第一个爬虫程序，参照别人的代码和讲解依葫芦画瓢，自己添加了对阅读次数的处理，后续准备对Scrapy爬虫源码进行阅读学习。

      参考链接：http://blog.csdn.net/u012150179/article/details/34486677