爬虫---Scrapy

Scrapy是一个为了爬取网站数据，提取结构性数据而编写的应用框架。 其可以应用在数据挖掘，信息处理或存储历史数据等一系列的程序中。
其最初是为了页面抓取 (更确切来说, 网络抓取 )所设计的， 也可以应用在获取API所返回的数据(例如 Amazon Associates Web Services ) 或者通用的网络爬虫。Scrapy用途广泛，可以用于数据挖掘、监测和自动化测试。

Scrapy 使用了 Twisted异步网络库来处理网络通讯。整体架构大致如下

 

 

Scrapy主要包括了以下组件：

• 引擎(Scrapy)
  用来处理整个系统的数据流处理, 触发事务(框架核心)
• 调度器(Scheduler)
  用来接受引擎发过来的请求, 压入队列中, 并在引擎再次请求的时候返回. 可以想像成一个URL（抓取网页的网址或者说是链接）的优先队列, 由它来决定下一个要抓取的网址是什么, 同时去除重复的网址
• 下载器(Downloader)
  用于下载网页内容, 并将网页内容返回给蜘蛛(Scrapy下载器是建立在twisted这个高效的异步模型上的)
• 爬虫(Spiders)
  爬虫是主要干活的, 用于从特定的网页中提取自己需要的信息, 即所谓的实体(Item)。用户也可以从中提取出链接,让Scrapy继续抓取下一个页面
• 项目管道(Pipeline)
  负责处理爬虫从网页中抽取的实体，主要的功能是持久化实体、验证实体的有效性、清除不需要的信息。当页面被爬虫解析后，将被发送到项目管道，并经过几个特定的次序处理数据。
• 下载器中间件(Downloader Middlewares)
  位于Scrapy引擎和下载器之间的框架，主要是处理Scrapy引擎与下载器之间的请求及响应。
• 爬虫中间件(Spider Middlewares)
  介于Scrapy引擎和爬虫之间的框架，主要工作是处理蜘蛛的响应输入和请求输出。
• 调度中间件(Scheduler Middewares)
  介于Scrapy引擎和调度之间的中间件，从Scrapy引擎发送到调度的请求和响应。

在Scrapy的数据流是由执行引擎控制，具体流程如下：

• 1、spiders产生request请求，将请求交给引擎
• 2、引擎(EGINE)吧刚刚处理好的请求交给了调度器，以一个队列或者堆栈的形式吧这些请求保存起来，调度一个出来再传给引擎
• 3、调度器(SCHEDULER)返回给引擎一个要爬取的url
• 4、引擎把调度好的请求发送给download，通过中间件发送（这个中间件至少有 两个方法，一个请求的，一个返回的)，
• 5、一旦完成下载就返回一个response，通过下载器中间件,返回给引擎，引擎把response 对象传给下载器中间件，最后到达引擎
• 6、引擎从下载器中收到response对象，从下载器中间件传给了spiders（spiders里面做两件事，1、产生request请求，2、为request请求绑定一个回调函数），spiders只负责解析爬取的任务。不做存储，
• 7、解析完成之后返回一个解析之后的结果items对象及(跟进的)新的Request给引擎
• 就被ITEM PIPELUMES处理了
• 8、引擎将(Spider返回的)爬取到的Item给Item Pipeline，存入数据库，持久化,如果数据不对，可重新封装成一个request请求，传给调度器
• 9、(从第二步)重复直到调度器中没有更多地request，引擎关闭该网站

 

Scrapy运行流程大概如下：

1. 引擎从调度器中取出一个链接(URL)用于接下来的抓取
2. 引擎把URL封装成一个请求(Request)传给下载器
3. 下载器把资源下载下来，并封装成应答包(Response)
4. 爬虫解析Response
5. 解析出实体（Item）,则交给实体管道进行进一步的处理
6. 解析出的是链接（URL）,则把URL交给调度器等待抓取

 

 scrapy框架分为七大部分核心的组件

1、引擎(EGINE)

引擎负责控制系统所有组件之间的数据流，并在某些动作发生时触发事件。有关详细信息，请参见上面的数据流部分。

2、调度器(SCHEDULER)

用来接受引擎发过来的请求, 压入队列中, 并在引擎再次请求的时候返回. 可以想像成一个URL的优先级队列, 由它来决定下一个要抓取的网址是什么, 同时去除重复的网址

3、下载器(DOWLOADER)

用于下载网页内容, 并将网页内容返回给EGINE，下载器是建立在twisted这个高效的异步模型上的

4、爬虫(SPIDERS)

SPIDERS是开发人员自定义的类，用来解析responses，并且提取items，或者发送新的请求

5、项目管道(ITEM PIPLINES)

在items被提取后负责处理它们，主要包括清理、验证、持久化（比如存到数据库）等操作

6、下载器中间件(Downloader Middlewares)

 下载器中间件是在引擎及下载器之间的特定钩子(specific hook)，处理Downloader传递给引擎的response。 其提供了一个简便的机制，通过插入自定义代码来扩展Scrapy功能。

7、爬虫中间件(Spider Middlewares)

 Spider中间件是在引擎及Spider之间的特定钩子(specific hook)，处理spider的输入(response)和输出(items及requests)。 其提供了一个简便的机制，通过插入自定义代码来扩展Scrapy功能。

 

一、安装

Linux
      pip3 install scrapy  # 安装前要安装twisted模块
 
 
Windows
      a. pip3 install wheel
      b. 下载twisted http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
      c. 进入下载目录，执行 pip3 install Twisted‑17.1.0‑cp35‑cp35m‑win_amd64.whl
      d. pip3 install scrapy
      e. 下载并安装pywin32：https://sourceforge.net/projects/pywin32/files/

 

 

二、基本使用

1. 基本命令

1. scrapy startproject 项目名称　　　　　　　　　　　　　　　　　　#　　Step1： 创建项目
   - 在当前目录中创建中创建一个项目文件（类似于Django）
 
2. scrapy genspider [-t template] <name> <domain>　　　　　　#    Step2： 创建爬虫
   - 创建爬虫应用
   cd 项目名称
   如：
      scrapy genspider -t basic oldboy oldboy.com
      scrapy genspider -t xmlfeed autohome autohome.com.cn
   PS:
      查看所有命令：scrapy gensipider -l
      查看模板命令：scrapy gensipider -d 模板名称
 
3. scrapy list
   - 展示爬虫应用列表
 
4. 先进入project，
        scrapy crawl 爬虫应用名称
   - 运行单独爬虫应用

命令行工具

#1 查看帮助
    scrapy -h
    scrapy <command> -h

#2 有两种命令：其中Project-only必须切到项目文件夹下才能执行，而Global的命令则不需要
    Global commands:
        startproject #创建项目
        genspider    #创建爬虫程序
        settings     #如果是在项目目录下，则得到的是该项目的配置
        runspider    #运行一个独立的python文件，不必创建项目
        shell        #scrapy shell url地址  在交互式调试，如选择器规则正确与否
        fetch        #独立于程单纯地爬取一个页面，可以拿到请求头
        view         #下载完毕后直接弹出浏览器，以此可以分辨出哪些数据是ajax请求
        version      #scrapy version 查看scrapy的版本，scrapy version -v查看scrapy依赖库的版本
    Project-only commands:
        crawl        #运行爬虫，必须创建项目才行，确保配置文件中ROBOTSTXT_OBEY = False
        check        #检测项目中有无语法错误
        list         #列出项目中所包含的爬虫名
        edit         #编辑器，一般不用
        parse        #scrapy parse url地址 --callback 回调函数  #以此可以验证我们的回调函数是否正确
        bench        #scrapy bentch压力测试

#3 官网链接
    https://docs.scrapy.org/en/latest/topics/commands.html

全局命令：所有文件夹都使用的命令,可以不依赖与项目文件也可以执行
项目的文件夹下执行的命令
1、scrapy startproject Myproject #创建项目
   cd Myproject
2、scrapy genspider baidu www.baidu.com  #创建爬虫程序，baidu是爬虫名,定位爬虫的名字
#写完域名以后默认会有一个url,
3、scrapy settings --get BOT_NAME  #获取配置文件
#全局：4、scrapy runspider budui.py
5、scrapy runspider AMAZON\spiders\amazon.py  #执行爬虫程序
   在项目下：scrapy crawl amazon  #指定爬虫名，定位爬虫程序来运行程序
    #robots.txt 反爬协议：在目标站点建一个文件，里面规定了哪些能爬，哪些不能爬
    # 有的国家觉得是合法的，有的是不合法的，这就产生了反爬协议
    # 默认是ROBOTSTXT_OBEY = True
    # 修改为ROBOTSTXT_OBEY = False  #默认不遵循反扒协议
6、scrapy shell https://www.baidu.com  #直接超目标站点发请求
     response
     response.status
     response.body
     view(response)
7、scrapy view https://www.taobao.com #如果页面显示内容不全，不全的内容则是ajax请求实现的，以此快速定位问题
8、scrapy version  #查看版本
9、scrapy version -v #查看scrapy依赖库锁依赖的版本
10、scrapy fetch --nolog http://www.logou.com    #获取响应的内容
11、scrapy fetch --nolog --headers http://www.logou.com  #获取响应的请求头
    (venv3_spider) E:\twisted\scrapy框架\AMAZON>scrapy fetch --nolog --headers http://www.logou.com
    > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    > Accept-Language: en
    > User-Agent: Scrapy/1.5.0 (+https://scrapy.org)
    > Accept-Encoding: gzip,deflate
    >
    < Content-Type: text/html; charset=UTF-8
    < Date: Tue, 23 Jan 2018 15:51:32 GMT
    < Server: Apache
    >代表请求
    <代表返回
10、scrapy shell http://www.logou.com #直接朝目标站点发请求
11、scrapy check  #检测爬虫有没有错误
12、scrapy list  #所有的爬虫名
13、scrapy parse http://quotes.toscrape.com/ --callback parse #验证回调函函数是否成功执行
14、scrapy bench #压力测试

 指定保存格式：

scrapy crawl quotes -o quotes.xml    

scrapy crawl quotes -o quotes.json

scrapy crawl quotes -o quotes.jl        # jsonline格式

scrapy crawl quotes -o quotes.csv         # 存储为csv格式

scrapy crawl quotes -o quotes.marshal    # marshal数据分析存储结构

还支持：scrapy crawl quotes -o ftp://user:pass@ftp.example.com/path/quotes.csv

 

 

2.项目结构以及爬虫应用简介

project_name/
   scrapy.cfg
   project_name/
       __init__.py
       items.py
       pipelines.py
       settings.py
       spiders/
           __init__.py
           爬虫1.py
           爬虫2.py
           爬虫3.py

文件说明：

• scrapy.cfg  项目的主配置信息。（真正爬虫相关的配置信息在settings.py文件中）
• items.py    设置数据存储模板，用于结构化数据，如：Django的Model
• pipelines    数据处理行为，如：一般结构化的数据持久化
• settings.py 配置文件，如：递归的层数、并发数，延迟下载等
• spiders      爬虫目录，如：创建文件，编写爬虫规则

注意：一般创建爬虫文件时，以网站域名命名

import scrapy
 
class XiaoHuarSpider(scrapy.spiders.Spider):
    name = "baidu"                            # 爬虫名称 *****
    allowed_domains = ["baidu.com"]  # 允许的域名
    start_urls = [
        "http://www.baidu.com/",   # 其实URL
    ]
 
    def parse(self, response):
        # 访问起始URL并获取结果后的回调函数

import sys,io
sys.stdout=io.TextIOWrapper(sys.stdout.buffer,encoding='gb18030')

报错信息：
UnicodeEncodeError: 'gbk' codec can't encode character '\u2764' in positiononse 

默认只能在cmd中执行爬虫，如果想在pycharm中执行需要做：
#在项目目录下新建：entrypoint.py
from scrapy.cmdline import execute
# execute(['scrapy', 'crawl', 'amazon','--nolog'])  #不要日志打印
# execute(['scrapy', 'crawl', 'amazon'])

#我们可能需要在命令行为爬虫程序传递参数，就用下面这样的命令
#acrapy crawl amzaon -a keyword=iphone8
execute(['scrapy', 'crawl', 'amazon1','-a','keyword=iphone8','--nolog'])  #不要日志打印

# execute(['scrapy', 'crawl', 'amazon1'])

 

3. Demo

# -*- coding: utf-8 -*-
import scrapy
from scrapy.selector import HtmlXPathSelector,Selector
from scrapy.http import Request

class XiaohuarSpider(scrapy.Spider):
    name = 'xiaohuar'
    allowed_domains = ['xiaohuar.com']
    start_urls = ['http://www.xiaohuar.com/hua/']

    def parse(self, response):　　　　# 回调函数
        # 要废弃
        # hxs = HtmlXPathSelector(response)
        # print(hxs)
        # result = hxs.select('//a[@class="item_list"]')

        hxs = Selector(response=response)
        # print(hxs)
        user_list = hxs.xpath('//div[@class="item masonry_brick"]')
        for item in user_list:
            price = item.xpath('./span[@class="price"]/text()').extract_first()
            url = item.xpath('div[@class="item_t"]/div[@class="class"]//a/@href').extract_first()
            print(price,url)

        result = hxs.xpath('/a[re:test(@href,"http://www.xiaohuar.com/list-1-\d+.html")]/@href')
        print(result)
        result = ['http://www.xiaohuar.com/list-1-1.html','http://www.xiaohuar.com/list-1-2.html']

        # 规则，递归
        for url in result:
            yield Request(url=url,callback=self.parse)                 # yield的是Request对象就会把函数放入scrapy的Scheduler调度器，递归爬取；其它对象会放入pipline进行持久化存储
            

执行此爬虫文件，则在终端进入项目目录执行如下命令：

scrapy crawl xiaohuar --nolog

对于上述代码重要之处在于：

• Request是一个封装用户请求的类，在回调函数中yield该对象表示继续访问
• HtmlXpathSelector用于结构化HTML代码并提供选择器功能

 

4. 选择器

scrapy shell https://doc.scrapy.org/en/latest/_static/selectors-sample1.html

<html>
 <head>
  <base href='http://example.com/' />
  <title>Example website</title>
 </head>
 <body>
  <div id='images'>
   <a href='image1.html'>Name: My image 1 <br /><img src='image1_thumb.jpg' /></a>
   <a href='image2.html'>Name: My image 2 <br /><img src='image2_thumb.jpg' /></a>
   <a href='image3.html'>Name: My image 3 <br /><img src='image3_thumb.jpg' /></a>
   <a href='image4.html'>Name: My image 4 <br /><img src='image4_thumb.jpg' /></a>
   <a href='image5.html'>Name: My image 5 <br /><img src='image5_thumb.jpg' /></a>
  </div>
 </body>
</html>

In [5]: response.selector.xpath('//title/text()').extract_first()
Out[5]: 'Example website'

In [8]: response.selector.css('title::text').extract_first()
Out[8]: 'Example website'

# selector可省略：
In [16]: response.css('title::text').extract_first()
Out[16]: 'Example website'

In [17]: response.xpath('//title/text()').extract_first()
Out[17]: 'Example website'

In [18]: response.xpath('//title/text()')
Out[18]: [<Selector xpath='//title/text()' data='Example website'>]




xpath中我们获取元素是通过.entry-header h1::text
如果是属性则用.entry-header a::attr(href) 



In [20]: response.xpath('//div[@id="images"]').css('img::attr(src)')            # '//div[@id="images"]')中，images外侧必须用双引号
Out[20]: 
[<Selector xpath='descendant-or-self::img/@src' data='image1_thumb.jpg'>,
 <Selector xpath='descendant-or-self::img/@src' data='image2_thumb.jpg'>,
 <Selector xpath='descendant-or-self::img/@src' data='image3_thumb.jpg'>,
 <Selector xpath='descendant-or-self::img/@src' data='image4_thumb.jpg'>,
 <Selector xpath='descendant-or-self::img/@src' data='image5_thumb.jpg'>]

In [23]: response.xpath('//div[@id="images"]').css('img::attr(src)').extract()
Out[23]: 
['image1_thumb.jpg',
 'image2_thumb.jpg',
 'image3_thumb.jpg',
 'image4_thumb.jpg',
 'image5_thumb.jpg']

In [21]: response.xpath('//div[@id="images"]').css('img::attr(src)').extract_first()
Out[21]: 'image1_thumb.jpg'



# default设置默认参数，当查找不到指定的attr时，用default代替，防止报错
In [25]: response.xpath('//div[@id="images"]').css('img::attr(srcc)').extract_first(default='replace')
Out[25]: 'replace'


# css、xpath获取属性：
In [27]: response.xpath('//a/@href')
Out[27]: 
[<Selector xpath='//a/@href' data='image1.html'>,
 <Selector xpath='//a/@href' data='image2.html'>,
 <Selector xpath='//a/@href' data='image3.html'>,
 <Selector xpath='//a/@href' data='image4.html'>,
 <Selector xpath='//a/@href' data='image5.html'>]

In [28]: response.xpath('//a/@href').extract()
Out[28]: ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']

In [29]: response.css('a::attr(href)')
Out[29]: 
[<Selector xpath='descendant-or-self::a/@href' data='image1.html'>,
 <Selector xpath='descendant-or-self::a/@href' data='image2.html'>,
 <Selector xpath='descendant-or-self::a/@href' data='image3.html'>,
 <Selector xpath='descendant-or-self::a/@href' data='image4.html'>,
 <Selector xpath='descendant-or-self::a/@href' data='image5.html'>]

In [30]: response.css('a::attr(href)').extract()
Out[30]: ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']



# css、xpath获取文本信息：
In [35]: response.css('a::text')
Out[35]: 
[<Selector xpath='descendant-or-self::a/text()' data='Name: My image 1 '>,
 <Selector xpath='descendant-or-self::a/text()' data='Name: My image 2 '>,
 <Selector xpath='descendant-or-self::a/text()' data='Name: My image 3 '>,
 <Selector xpath='descendant-or-self::a/text()' data='Name: My image 4 '>,
 <Selector xpath='descendant-or-self::a/text()' data='Name: My image 5 '>]

In [36]: response.css('a::text').extract_first()
Out[36]: 'Name: My image 1 '


In [37]: response.xpath('//a/text()')
    ...: 
Out[37]: 
[<Selector xpath='//a/text()' data='Name: My image 1 '>,
 <Selector xpath='//a/text()' data='Name: My image 2 '>,
 <Selector xpath='//a/text()' data='Name: My image 3 '>,
 <Selector xpath='//a/text()' data='Name: My image 4 '>,
 <Selector xpath='//a/text()' data='Name: My image 5 '>]

In [38]: response.xpath('//a/text()').extract_first()
Out[38]: 'Name: My image 1 '



# css、xpath 选取属性包含指定名字的标签：
In [42]: response.xpath('//a[contains(@href, "image")]/@href')
Out[42]: 
[<Selector xpath='//a[contains(@href, "image")]/@href' data='image1.html'>,
 <Selector xpath='//a[contains(@href, "image")]/@href' data='image2.html'>,
 <Selector xpath='//a[contains(@href, "image")]/@href' data='image3.html'>,
 <Selector xpath='//a[contains(@href, "image")]/@href' data='image4.html'>,
 <Selector xpath='//a[contains(@href, "image")]/@href' data='image5.html'>]


In [46]: response.css('a[href*="image"]::attr(href)')
Out[46]: 
[<Selector xpath="descendant-or-self::a[@href and contains(@href, 'image')]/@href" data='image1.html'>,
 <Selector xpath="descendant-or-self::a[@href and contains(@href, 'image')]/@href" data='image2.html'>,
 <Selector xpath="descendant-or-self::a[@href and contains(@href, 'image')]/@href" data='image3.html'>,
 <Selector xpath="descendant-or-self::a[@href and contains(@href, 'image')]/@href" data='image4.html'>,
 <Selector xpath="descendant-or-self::a[@href and contains(@href, 'image')]/@href" data='image5.html'>]

 
# css选择器http://www.w3school.com.cn/cssref/css_selectors.asp：    
[attribute^=value]    a[src^="https"]    选择其 src 属性值以 "https" 开头的每个 <a> 元素。    
[attribute$=value]    a[src$=".pdf"]    选择其 src 属性以 ".pdf" 结尾的所有 <a> 元素。    
[attribute*=value]    a[src*="abc"]    选择其 src 属性中包含 "abc" 子串的每个 <a> 元素。



# 级联获取
# xpath方法：
In [49]: response.xpath('//a[contains(@href, "image")]/img/@src').extract()
Out[49]: 
['image1_thumb.jpg',
 'image2_thumb.jpg',
 'image3_thumb.jpg',
 'image4_thumb.jpg',
 'image5_thumb.jpg']
 
# CSS方法:
In [54]: response.css('a[href*=image] img::attr(src)').extract()
Out[54]: 
['image1_thumb.jpg',
 'image2_thumb.jpg',
 'image3_thumb.jpg',
 'image4_thumb.jpg',
 'image5_thumb.jpg']
 

 
 
# css、xpath 的 re正则：
In [55]: response.css('a::text').re('Name:(.*)')
Out[55]: 
[' My image 1 ',
 ' My image 2 ',
 ' My image 3 ',
 ' My image 4 ',
 ' My image 5 ']

In [56]: response.xpath('//a/text()').re('Name:(.*)')
Out[56]: 
[' My image 1 ',
 ' My image 2 ',
 ' My image 3 ',
 ' My image 4 ',
 ' My image 5 ']

# re_first取匹配结果的第一个
In [61]: response.css('a::text').re_first('Name:(.*)').strip()
Out[61]: 'My image 1'

In [59]: response.xpath('//a/text()').re_first('Name:(.*)').strip()
Out[59]: 'My image 1'

 

#!/usr/bin/env python
# -*- coding:utf-8 -*-
from scrapy.selector import Selector, HtmlXPathSelector
from scrapy.http import HtmlResponse


html = """<!DOCTYPE html>
<html>
    <head lang="en">
        <meta charset="UTF-8">
        <title></title>
    </head>
    <body>
        <ul>
            <li class="item-"><a id='i1' href="link.html">first item</a></li>
            <li class="item-0"><a id='i2' href="llink.html">first item</a></li>
            <li class="item-1"><a href="llink2.html">second item<span>vv</span></a></li>
        </ul>
        <div><a href="llink2.html">second item</a></div>
    </body>
</html>
"""
response = HtmlResponse(url='http://example.com', body=html, encoding='utf-8')
hxs = HtmlXPathSelector(response)
print(hxs)
hxs = Selector(response=response).xpath('//a')      # 找html页面中全部a标签
for i in hxs:
    print(i)
print(hxs)

hxs = Selector(response=response).xpath('//a[2]') # 结果为空？
for i in hxs:
    print(i)
print(hxs)
print(hxs)

hxs = Selector(response=response).xpath('//a[@id]')     # 找具有id属性的a标签
print(hxs)

hxs = Selector(response=response).xpath('//a[@id="i1"]')   # 找具有id属性且值为i1的a标签
print(hxs)

hxs = Selector(response=response).xpath('//a[@href="link.html"][@id="i1"]') # href属性值为link.html且id属性且值为i1的a标签
print(hxs)

hxs = Selector(response=response).xpath('//a[contains(@href, "link2")]')     # contains函数：匹配一个属性值中包含的字符串，找出属性href值包含link2的a标签
print(hxs)

hxs = Selector(response=response).xpath('//a[starts-with(@href, "link")]')  # 属性href以link开头的a标签
print(hxs)

hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]')        # 正则匹配，id属性包含“i+数字”的a标签
print(hxs)
hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]/text()').extract()   # id属性包含“i+数字”的a标签的值
print(hxs)
hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]/@href').extract()  # id属性包含“i+数字”的a标签的href属性值
print(hxs)
hxs = Selector(response=response).xpath('/html/body/ul/li/a/@href').extract()           # 取/html/body/ul/li/a/路径下@href的值，多个值返回列表
print(hxs)
hxs = Selector(response=response).xpath('//body/ul/li/a/@href').extract_first()# 取/html/body/ul/li/a/路径下@href的第一个值
print(hxs)


#  相对路径的xpath
ul_list = Selector(response=response).xpath('//body/ul/li')
for item in ul_list:
    # v = item.xpath('./a/span')
    # 或
    # v = item.xpath('a/span')
    # 或
    v = item.xpath('*/a/span')
    print(v)

for item in obj:
    price = item.xpath('.//span[@class="price"]/text()').extract_first()
    url = item.xpath('div[@class="item_t"]/div[@class="class"]//a/@href').extract_first()

# 一般情况：
     // 表示寻找范围为：所有子孙
     /  表示寻找范围为：儿子

# 特殊：
    item.xpath('./')  # 相对当前子孙中找
    item.xpath('a')  # 相对当前子孙中找


# 几个简单的例子：
　　/html/head/title: 选择HTML文档<head>元素下面的<title> 标签。
　　/html/head/title/text(): 选择前面提到的<title> 元素下面的文本内容;
　　//td: 选择所有 <td> 元素;
　　//div[@class="mine"]: 选择所有包含 class="mine" 属性的div 标签元素;

基本的路径意义：

　

表达式	描述
nodename	选取此节点的所有子节点。
/	从根节点选取。
//	从匹配选择的当前节点选择文档中的节点，而不考虑它们的位置。
.	选取当前节点。
..	选取当前节点的父节点。
@	选取属性。

　　

注意：settings.py中设置DEPTH_LIMIT = 1来指定“递归”的层数。

 

 

 

三.自定制scrapy

1. 起始URL - parse

import scrapy
from scrapy.http import Request

class ChoutiSpider(scrapy.Spider):
    name = 'chouti'
    allowed_domains = ['chouti.com']
    start_urls = ['http://chouti.com/']            # 起始url

    def start_requests(self):                    # 重定义Spider类中start_requests方法，进而可以自定义回调函数名
        for url in self.start_urls:
            yield Request(url, dont_filter=True,callback=self.parse1)

    def parse1(self, response):
        pass

 

2. POST请求，请求头

回顾：requests模块发送get或post请求：

requests.get(params={},headers={},cookies={})
requests.post(params={},headers={},cookies={},data={},json={})

url, 
method='GET', 
headers=None, 
body=None,
cookies=None,

 

在scrapy中发送get或post请求参数说明：

# GET请求:
url, 
method='GET', 
headers={}, 
cookies={}, cookiejar

# POST请求:
url, 
method='GET', 
headers={}, 
cookies={}, cookiejar
body=None,

# application/x-www-form-urlencoded; charset=UTF-8                # 对于这种请求头的Content-Type，需要这种arg1=66&arg2=66&oneMonth=1字符串格式
    form_data = {
        'user':'alex',
        'pwd': 123
    }
    import urllib.parse    
    data = urllib.parse.urlencode({'k1':'v1','k2':'v2'})        # 把字典转换为arg1=66&arg2=66&oneMonth=1字符串格式
    
    "phone=86155fa&password=asdf&oneMonth=1"   
    
# application/json; charset=UTF-8                # 对于这种请求头的Content-Type，直接把字典json.dumps()即可
    json.dumps()
    
    "{k1:'v1','k2':'v2'}"   

示例：
     Request(
        url='http://dig.chouti.com/login',
        method='POST',
        headers={'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'},
        body='phone=8615131255089&password=pppppppp&oneMonth=1',
        callback=self.check_login
    )

 

 

3.cookie

通过cookie_jar获取response返回中的cookie
第一种方式：通过把cookie放入构造cookie_dict字典发送

def login(self, response):
    cookie_dict = {}
    cookie_jar = CookieJar()　　　　　　　　　　　　　　　　　　　　# 创建cookie_jar对象
    cookie_jar.extract_cookies(response, response.request)　　# 利用cookie_jar提取响应中的cookie
    for k, v in cookie_jar._cookies.items():
        for i, j in v.items():
            for m, n in j.items():
                self.cookie_dict[m] = n.value

    req = Request(
        url='http://dig.chouti.com/login',
        method='POST',
        headers={'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'},
        body='phone=8615131255089&password=pppppppp&oneMonth=1',
        cookies=self.cookie_dict,　　　　　　# 以字典的形式封装氢气
        callback=self.check_login
    )
    yield req

 

第二种方式(推荐)：直接携带cookie_jar对象到Request中发送

def login(self, response):
    cookie_jar = CookieJar()
    cookie_jar.extract_cookies(response, response.request)
    
    req = Request(
        url='http://dig.chouti.com/login',
        method='POST',
        headers={'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'},
        body='phone=8615131255089&password=pppppppp&oneMonth=1',
        cookies=self.cookie_jar,　　　　# 以对象的形式封装请求
        callback=self.check_login
    )
    yield req

 

class ChoutiSpider(scrapy.Spider):
    name = 'chouti'
    allowed_domains = ['chouti.com']
    start_urls = ['http://chouti.com/']

    def start_requests(self):
        for url in self.start_urls:
            yield Request(url, dont_filter=True, callback=self.parse1)

    def parse1(self, response):
        """获取首页登录"""
        # response.text 首页所有内容
        from scrapy.http.cookies import CookieJar
        cookie_jar = CookieJar()
        self.cookie_jar = cookie_jar.extract_cookies(response, response.request)  # 获取响应中的cookies

        post_dict = {
            'phone': '8617748232617',
            'password': 'password',
            'oneMonth': 1,
        }

        import urllib.parse
        data = urllib.parse.urlencode(post_dict)    # urlencode转换为：phone=86123&password=123&oneMonth=1这种格式
        # 发送post请求准备登录
        yield Request(
            url='http://dig.chouti.com/login',
            method='POST',
            cookies=self.cookie_jar,
            body=data,
            headers={'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'},
            callback=self.parse2
        )

    def parse2(self, response):
        """response返回登录结果"""
        print(response.text)
        #  获取新闻列表
        yield Request(url='http://dig.chouti.com', cookies=self.cookie_jar, callback=self.parse3)

    def parse3(self, response):
        """点赞"""
        hxs = Selector(response)
        linkid_list = hxs.xpath('//div[@class="news-pic"]/img/@lang').extract()
        print(linkid_list)
        for link_id in linkid_list:
            # 获取每一个id去点赞
            base_url = "https://dig.chouti.com/link/vote?linksId={0}".format(link_id)
            yield Request(url=base_url, method='POST', cookies=self.cookie_jar, callback=self.parse4)

        # hxs.xpath('//div[@id="dig_lcpage"]//a/@href')
        # 寻找所有分页页面
        page_list = hxs.xpath('//a[@class="ct_pagepa"] /@href').extract()
        """https://dig.chouti.com/all/hot/recent/2"""
        for page in page_list:
            page_url = "https://dig.chouti.com%s" % page
            yield Request(url=page_url, method='GET', cookies=self.cookie_jar, callback=self.parse3)

    def parse4(self, responese):
        # 输出点赞结果
        print(responese.text)

 

pipeline持久化写入文件

# jandan.py:
#!/usr/bin/env python
# -*-coding:utf-8 -*-
import scrapy
from scrapy.selector import HtmlXPathSelector, Selector
from scrapy.http.request import Request


class JianDanSpider(scrapy.Spider):
    name = 'jiandan'
    allowed_domains = ['jandan.net']
    start_urls = ['http://jandan.net/']

    def start_requests(self):
        for url in self.start_urls:
            yield Request(url, dont_filter=True, callback=self.parse1)

    def parse1(self, response):
        # print(response.text)
        hxs = Selector(response)
        a_list = hxs.xpath('//div[@class="indexs"]/h2')
        for tag in a_list:
            url = tag.xpath('./a/@href').extract_first()
            text = tag.xpath('./a/text()').extract_first()
            from ..items import Sp2Item
            yield Sp2Item(url=url, text=text)             # 会执行pipeline中的process_item方法



# pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html

from scrapy.exceptions import DropItem


class Sp2Pipeline(object):
    def __init__(self, val):
        self.val = val
        self.f = None

    def process_item(self, item, spider):
        """
        :param item:  爬虫中yield回来的对象，是一个一个返回的，spider中循环一次返回一个item
        :param spider: 爬虫对象 obj = JianDanSpider(); 有spider.name等属性
        """
        a = item['url'] + ' ' + item['text'] + '\n'
        self.f.write(a)        # 持久化写入文件
        return item

    @classmethod
    def from_crawler(cls, crawler):
        """
        用于实例化Sp2Pipeline对象，过程中可以通过crawler封装一些参数到对象中
        初始化时候，用于创建pipeline对象,classmethod装饰器会把cls替换为当前类
        :param crawler:有爬虫相关的参数封装，可以取配置文件中的值（settings定义的key必须大写）：crawler.settings.get('MMMM')
        :return:
        """
        print('执行pipeline的from_crawler方法，进行实例化对象')
        val = crawler.settings.get('MMMM')
        return cls(val)  # 实例化对象，并通过对象的__init__方法初始化val封装到对象中

    def open_spider(self, spider):
        """
        爬虫开始执行时，调用
        :param spider:
        :return:
        """
        self.f = open('a.log', 'a+', encoding='utf-8')
        print('打开爬虫')

    def close_spider(self,spider):
        """
        爬虫关闭时，被调用
        :param spider:
        :return:
        """
        self.f.close()
        print('关闭爬虫')



items.py:
# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class Sp2Item(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    """负责结构化、规则化数据字段"""
    url = scrapy.Field()
    text = scrapy.Field()
   

多个pipeline执行顺序说明：

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html


class Sp2Pipeline(object):
    def __init__(self, val):
        self.val = val
        self.f = None

    def process_item(self, item, spider):
        """
        :param item:  爬虫中yield回来的对象
        :param spider: 爬虫对象 obj = JianDanSpider(); 有spider.name等属性
        """
        if spider.name == 'jiadnan':        # 由于pipeline是全局的，所有的spider执行时，只要yield过来item对象，都会执行pipeline，这里可以通过spider.name属性做判断，选择性的执行pipeline
            pass
        print(item)

        a = item['url'] + ' ' + item['text'] + '\n'
        self.f.write(a)
        # return item             # 将item传递给下一个pipeline的process_item方法，不写传递None，依然会执行
        # from scrapy.exceptions import DropItem
        # raise DropItem()          # DropItem:下一个pipeline的process_item方法不在执行

    @classmethod
    def from_crawler(cls, crawler):
        """
        用于实例化Sp2Pipeline对象，过程中可以通过crawler封装一些参数到对象中
        初始化时候，用于创建pipeline对象,classmethod装饰器会把cls替换为当前类
        :param crawler:有爬虫相关的参数封装，可以取配置文件中的值（settings定义的key必须大写）：crawler.settings.get('MMMM')
        :return:
        """
        print('执行pipeline的from_crawler方法，进行实例化对象')
        val = crawler.settings.get('MMMM')
        return cls(val)  # 实例化对象，并通过对象的__init__方法初始化val封装到对象中

    def open_spider(self, spider):
        """
        爬虫开始执行时，调用
        :param spider:
        :return:
        """
        self.f = open('a.log', 'a+', encoding='utf-8')
        print('打开爬虫')

    def close_spider(self,spider):
        """
        爬虫关闭时，被调用
        :param spider:
        :return:
        """
        self.f.close()
        print('关闭爬虫')


class Sp3Pipeline(object):
    def __init__(self):
        self.f = None

    def process_item(self, item, spider):
        """

        :param item:  爬虫中yield回来的对象
        :param spider: 爬虫对象 obj = JianDanSpider()
        :return:
        """
        print('in the pipeline3--------------------------')
        return item

    @classmethod
    def from_crawler(cls, crawler):
        """
        初始化时候，用于创建pipeline对象
        :param crawler:
        :return:
        """
        # val = crawler.settings.get('MMMM')
        print('执行pipeline的from_crawler，进行实例化对象')
        return cls()

    def open_spider(self, spider):
        """
        爬虫开始执行时，调用
        :param spider:
        :return:
        """
        print('in the pipeline3-------------------------- 打开爬虫')
        # self.f = open('a.log', 'a+')

    def close_spider(self, spider):
        """
        爬虫关闭时，被调用
        :param spider:
        :return:
        """
        print('in the pipeline3-------------------------- 关闭爬虫')

 

小结：

为什么使用pipeline：
爬取的数据保存时，如果在spider中保存，会有频繁的打开/关闭文件、DB连接等等操作，放在pipeline中，只在爬虫运行前打开一次文件/DB连接即可

pipeline执行的前提：
- spider中yield Item（url=url,text='asdasd'）对象后(yield Item(),意为：持久化Item对象，交给pipeline处理)，在items中定义Item类，并在spider中导入后，会直接交给pipeline处理
- settings中注册
    ITEM_PIPELINES = {
       'sp2.pipelines.Sp2Pipeline': 300,        # 数字越小代表优先级越高
       'sp2.pipelines.Sp2Pipeline': 100,
    }

    
# 自定义pipeline的执行流程：

一：爬虫运行前的动作：
1.检测自定义的CustomPipeline类中是否有负责初始化的from_crawler方法(通过hasattr，getattr)
如果有：
       obj = 类.from_crawler()  #初始化时候，用于创建pipeline对象
如果没有：
       obj = 类()
       
2.打开爬虫，爬虫开爬前的一些动作写在这里：
obj.open_spider()
# 这里可以执行打开文件、数据库等待操作

二.执行调度器中的爬虫，process_item方法
while True:
    爬虫运行，并且执行parse各种各样的方法，yield item
    obj.process_item()

三. 如果存在更多的pipeline，依次执行他们的process_item方法，pipeline中的其他方法：open_spider等等不会执行；
如果不想继续执行pipeline：
from scrapy.exceptions import DropItem
raise DropItem()          # DropItem:下一个pipeline的process_item方法不在执行
    
四.关闭爬虫    
3.obj.close_spider()    


注意：
pipeline是全局的，所有的spider爬虫执行时，只要yield过来item对象，都会执行pipeline，这里可以通过spider.name属性做判断，选择性的执行pipeline

 补充：

scrapy：中文文档
http://scrapy-chs.readthedocs.io/zh_CN/latest/
英文：
https://docs.scrapy.org/en/latest/topics/spiders.html


allowed_domains：
每次生成一个Request时，判断当前url是否匹配allowed_domains指定的域名，匹配则爬取，否则丢弃；
包含了spider允许爬取的域名(domain)列表(list)。 当 OffsiteMiddleware 启用时， 域名不在列表中的URL不会被跟进。

custom_settings：
spiders中以类属性且为字典形式（键名：settings中的变量名）局部自定义的配置，会覆盖settings中的全局配置，可以设置spiders各自的useragent
A dictionary of settings that will be overridden from the project wide configuration when running this spider. It must be defined as a class attribute since the settings are updated before instantiation.


from_crawler(crawler, *args, **kwargs)
可以在spiders中定义，也可以在pipeline中定义；作用：利用它的crawler参数，可以通过settings中配置参数来构造类（通过crawler对象，可以获取scrapy所有核心组件，如全局配置的每个信息）

This is the class method used by Scrapy to create your spiders.
You probably won’t need to override this directly, since the default implementation acts as a proxy to the __init__() method, calling it with the given arguments args and named arguments kwargs.
Nonetheless, this method sets the crawler and settings attributes in the new instance, so they can be accessed later inside the spider’s code.

参数:    
crawler (Crawler instance) – crawler to which the spider will be bound
args (list) – arguments passed to the __init__() method
kwargs (dict) – keyword arguments passed to the __init__() method

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['quotes.toscrape.com']
    start_urls = ['http://quotes.toscrape.com/']


    def __init__(self, mongo_url, mongo_db, *args, **kwargs):
        super(QuotesSpider, self).__init__(*args, **kwargs)
        self.mongo_url = mongo_url
        self.mongo_db = mongo_db

    @classmethod
    def from_crawler(cls, crawler):
        print('in MongoPipeline from_crawler ---------------------------------')            # step 1
        return cls(
            mongo_url=crawler.settings.get('MONGO_URL'),
            mongo_db=crawler.settings.get('MONGO_DB')
        )
        

        
# spider的日志：        
Logging from Spiders
Scrapy provides a logger within each Spider instance, which can be accessed and used like this:
import scrapy

class MySpider(scrapy.Spider):

    name = 'myspider'
    start_urls = ['https://scrapinghub.com']

    def parse(self, response):
        self.logger.info('Parse function called on %s', response.url)    
        
        
That logger is created using the Spider’s name, but you can use any custom Python logger you want. For example:
import logging
import scrapy

logger = logging.getLogger('mycustomlogger')

class MySpider(scrapy.Spider):

    name = 'myspider'
    start_urls = ['https://scrapinghub.com']

    def parse(self, response):
        logger.info('Parse function called on %s', response.url)
        
        
        
        
pipeline: 可用来数据清洗、存储

# https://docs.scrapy.org/en/latest/topics/item-pipeline.html#

# from_crawler方法：
# 一般用他来获取scrapy的settings配置信息
from_crawler(cls, crawler)        
If present, this classmethod is called to create a pipeline instance from a Crawler. It must return a new instance of the pipeline. Crawler object provides access to all Scrapy core components like settings and signals; it is a way for pipeline to access them and hook its functionality into Scrapy.
Parameters:    crawler (Crawler object) – crawler that uses this pipeline



Item pipeline example
例1：
Price validation and dropping items with no prices
Let’s take a look at the following hypothetical pipeline that adjusts the price attribute for those items that do not include VAT (price_excludes_vat attribute), and drops those items which don’t contain a price:

from scrapy.exceptions import DropItem

class PricePipeline(object):

    vat_factor = 1.15

    def process_item(self, item, spider):
        if item['price']:
            if item['price_excludes_vat']:
                item['price'] = item['price'] * self.vat_factor
            return item
        else:
            raise DropItem("Missing price in %s" % item)

            
例2：            
Write items to a JSON file        
The following pipeline stores all scraped items (from all spiders) into a single items.jl file, containing one item per line serialized in JSON format:
import json

class JsonWriterPipeline(object):

    def open_spider(self, spider):
        self.file = open('items.jl', 'w')

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        line = json.dumps(dict(item)) + "\n"
        self.file.write(line)
        return item
注意：
The purpose of JsonWriterPipeline is just to introduce how to write item pipelines. If you really want to store all scraped items into a JSON file you should use the Feed exports：https://docs.scrapy.org/en/latest/topics/feed-exports.html#topics-feed-exports



例3：存储到MongoDB：
Write items to MongoDB
In this example we’ll write items to MongoDB using pymongo. MongoDB address and database name are specified in Scrapy settings; MongoDB collection is named after item class.

The main point of this example is to show how to use from_crawler() method and how to clean up the resources properly.:
pymongo官方api：https://api.mongodb.com/python/current/api/index.html
import pymongo

class MongoPipeline(object):

    collection_name = 'scrapy_items'

    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get('MONGO_URI'),
            mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')
        )

    def open_spider(self, spider):
        self.client = pymongo.MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]

    def close_spider(self, spider):
        self.client.close()

    def process_item(self, item, spider):
        self.db[self.collection_name].insert_one(dict(item))
        return item

例4：pipeline中去重
Duplicates filter
A filter that looks for duplicate items, and drops those items that were already processed. Let’s say that our items have a unique id, but our spider returns multiples items with the same id:
from scrapy.exceptions import DropItem

class DuplicatesPipeline(object):

    def __init__(self):
        self.ids_seen = set()

    def process_item(self, item, spider):
        if item['id'] in self.ids_seen:
            raise DropItem("Duplicate item found: %s" % item)
        else:
            self.ids_seen.add(item['id'])
            return item
            

 

from twisted.enterprise import adbapi



class MySQLPipeline(object):
    @classmethod
    def from_crawler(cls, crawler):
        # 从项目的配置文件中读取相应的参数
        cls.MYSQL_DB_NAME = crawler.settings.get("MYSQL_DB_NAME", 'scrapy_default')
        cls.HOST = crawler.settings.get("MYSQL_HOST", 'localhost')
        cls.PORT = crawler.settings.get("MYSQL_PORT", 3306)
        cls.USER = crawler.settings.get("MYSQL_USER", 'root')
        cls.PASSWD = crawler.settings.get("MYSQL_PASSWORD", '123456')
        return cls()

    def open_spider(self, spider):
        self.dbpool = adbapi.ConnectionPool('pymysql', host=self.HOST, port=self.PORT, user=self.USER,
                                            passwd=self.PASSWD, db=self.MYSQL_DB_NAME, charset='utf8')

    def close_spider(self, spider):
        self.dbpool.close()

    def process_item(self, item, spider):
        self.dbpool.runInteraction(self.insert_db, item)
        # query.addErrback(self.handle_error)
        return item

    def insert_db(self, tx, item):
        values = (
            item['houseRecord'], item['price'], item['unitPrice'], item['room'], item['type'], item['area'],
            item['communityName'], item['areaName'], item['visitTime'], str(item['base']), str(item['transaction']),
            item['base_more'], item['tags'], item['url']
        )
        sql = 'INSERT INTO lianjia(houseRecord, price,unitPrice,room,type,area,communityName,areaName,visitTime,' \
              'base,transaction,base_more,tags,url) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)'
        tx.execute(sql, values)

    # def handle_error(self, failure):
    #     print(failure)

# pipelines.py
import pymysql

class MysqlPipeline(object):
    def __init__(self, host, database, user, password, port):
        self.host = host
        self.database = database
        self.user = user
        self.password = password
        self.port = port

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            host=crawler.settings.get('MYSQL_HOST'),
            database=crawler.settings.get('MYSQL_DB'),
            user=crawler.settings.get('MYSQL_USER'),
            password=crawler.settings.get('MYSQL_PASSWORD'),
            port=crawler.settings.get('MYSQL_PORT')
        )

    def open_spider(self, spider):
        self.db = pymysql.connect(self.host,self.user,self.password, self.database, charset='utf8', port=self.port)
        self.cursor = self.db.cursor()


    def process_item(self, item, spider):
        data = dict(item)
        keys = ', '.join(data.keys())
        values = ', '.join(['%s'] * len(data))
        sql = 'insert into %s (%s) values (%s)' % (item.table, keys, values)
        self.cursor.execute(sql, tuple(data.values()))
        self.db.commit()
        return item


    def close_spider(self, spider):
        self.db.close()


# settings
MYSQL_HOST = '192.168.1.110'
MYSQL_DB = 'images360'
MYSQL_PORT = 3306      # 注意：port一定要是int类型，不能加引号
MYSQL_USER = 'root'
MYSQL_PASSWORD = '123456'
    

from twisted.enterprise import adbapi
class WebcrawlerScrapyPipeline(object):
    '''保存到数据库中对应的class
       1、在settings.py文件中配置
       2、在自己实现的爬虫类中yield item,会自动执行'''

    def __init__(self, dbpool):
        self.dbpool = dbpool

    @classmethod
    def from_settings(cls, settings):
        '''1、@classmethod声明一个类方法，而对于平常我们见到的叫做实例方法。
           2、类方法的第一个参数cls（class的缩写，指这个类本身），而实例方法的第一个参数是self，表示该类的一个实例
           3、可以通过类来调用，就像C.f()，相当于java中的静态方法'''
        #读取settings中配置的数据库参数
        dbparams = dict(
            host=settings['MYSQL_HOST'],  
            db=settings['MYSQL_DBNAME'],
            user=settings['MYSQL_USER'],
            passwd=settings['MYSQL_PASSWD'],
            charset='utf8',  # 编码要加上，否则可能出现中文乱码问题
            cursorclass=MySQLdb.cursors.DictCursor,
            use_unicode=False,
        )
        dbpool = adbapi.ConnectionPool('MySQLdb', **dbparams)  # **表示将字典扩展为关键字参数,相当于host=xxx,db=yyy....
        return cls(dbpool)  # 相当于dbpool付给了这个类，self中可以得到

    # pipeline默认调用
    def process_item(self, item, spider):
        query = self.dbpool.runInteraction(self._conditional_insert, item)  # 调用插入的方法
        query.addErrback(self._handle_error, item, spider)  # 调用异常处理方法
        return item

    # 写入数据库中
    # SQL语句在这里
    def _conditional_insert(self, tx, item):
        sql = "insert into jsbooks(author,title,url,pubday,comments,likes,rewards,views) values(%s,%s,%s,%s,%s,%s,%s,%s)"
        params = (item['author'], item['title'], item['url'], item['pubday'],item['comments'],item['likes'],item['rewards'],item['reads'])
        tx.execute(sql, params)

    # 错误处理方法
    def _handle_error(self, failue, item, spider):
        print failue

# https://blog.csdn.net/xdl1278/article/details/79056380        
from twisted.enterprise import adbapi
from MySQLdb import cursors

class MySQL_Twisted_Pipelines(object):
    # 基于twisted.enterprise.adbapi的异步MySQL管道
    # def __init__(self, dbpool):
    #     self.dbpool = dbpool

    # @classmethod
    # def from_settings(cls, settings):
    #     dbpool = adbapi.ConnectionPool("MySQLdb", **settings["MYSQL_INFO"],           cursorclass=cursors.DictCursor)
    #     return cls(dbpool)

    # @classmethod
    # def from_crawler(cls, crawler):
    #     settings = crawler.settings
    #     dbpool = adbapi.ConnectionPool("MySQLdb", **settings["MYSQL_INFO"], cursorclass=cursors.DictCursor)
    #     return cls(dbpool)

    def open_spider(self, spider):
        self.dbpool = adbapi.ConnectionPool("MySQLdb", **spider.settings["MYSQL_INFO"],  cursorclass=cursors.DictCursor)

    def process_item(self, item, spider):
        query = self.dbpool.runInteraction(self.db_insert, item)
        query.addErrback(self.handle_error)

    def handle_error(self, failure):
        print(failure)

    def db_insert(self, cursor, item):
        sql_insert = 'INSERT INTO teachers(name,title,info) VALUES (%s,%s,%s)'
        cursor.execute(sql_insert, (item['name'], item['title'], item['info']))

# 伪装浏览器请求，设置延迟抓取，防ban
DOWNLOAD_DELAY = 0.25 # 250 ms of delay

# 安装MySQLdb
pip3 install mysqlclient

# pipeline：
from scrapy import Request
from scrapy.exceptions import DropItem
from scrapy.pipelines.images import ImagesPipeline


class ImagePipeline(ImagesPipeline):
    def file_path(self, request, response=None, info=None):
        url = request.url
        file_name = url.split('/')[-1]
        return file_name

    def item_completed(self, results, item, info):
        image_paths = [x['path'] for ok, x in results if ok]
        if not image_paths:
            raise DropItem('Image Downloaded Failed')
        return item

    def get_media_requests(self, item, info):
        yield Request(item['url'])


# setting
IMAGES_STORE = './images'

# 开启
ITEM_PIPELINES = {
   'images360.pipelines.ImagePipeline': 300,

}

 

 

 

 

去重之避免重复访问url：

scrapy默认使用 scrapy.dupefilter.RFPDupeFilter 进行去重，相关配置有：

DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter'
DUPEFILTER_DEBUG = False　　　　　　　　　　　　　　　　　　　　
JOBDIR = "保存范文记录的日志路径，如：/root/"  # 最终路径为 /root/requests.seen

第2、3两行的配置是属于RFPDupeFilter 类独有的，自定义去重时，可以根据需要更改：在项目目录下自定义去重文件re.py，在settings中声明；

class RepeatUrl:
    def __init__(self):
        self.visited_url = set()        # 放在当前服务的内存，也可以放在memcache、redis中

    @classmethod
    def from_settings(cls, settings):
        """
        初始化时，调用
        :param settings: 
        :return: 
        """
        return cls()

    def request_seen(self, request):
        """
        检测当前请求是否已经被访问过
        :param request: 
        :return: True表示已经访问过；False表示未访问过
        """
        if request.url in self.visited_url:
            return True
        self.visited_url.add(request.url)
        return False

    def open(self):
        """
        开始爬去请求时，调用
        :return: 
        """
        print('open replication')

    def close(self, reason):
        """
        结束爬虫爬取时，调用
        :param reason: 
        :return: 
        """
        print('close replication')

    def log(self, request, spider):
        """
        记录日志
        :param request: 
        :param spider: 
        :return: 
        """
        print('repeat', request.url)
# 配置文件中定义：
DUPEFILTER_CLASS = 'sp2.rep.RepeatUrl'        

去重规则应该多个爬虫共享的，但凡一个爬虫爬取了，其他都不要爬了，实现方式如下

#方法一：
1、新增类属性
visited=set() #类属性

2、回调函数parse方法内：
def parse(self, response):
    if response.url in self.visited:
        return None
    .......

    self.visited.add(response.url) 

#方法一改进：针对url可能过长，所以我们存放url的hash值
def parse(self, response):
        url=md5(response.request.url)
    if url in self.visited:
        return None
    .......

    self.visited.add(url) 

#方法二：Scrapy自带去重功能
配置文件：
DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter' #默认的去重规则帮我们去重，去重规则在内存中
DUPEFILTER_DEBUG = False
JOBDIR = "保存范文记录的日志路径，如：/root/"  # 最终路径为 /root/requests.seen，去重规则放文件中

scrapy自带去重规则默认为RFPDupeFilter，只需要我们指定
Request(...,dont_filter=False) ，如果dont_filter=True则告诉Scrapy这个URL不参与去重。

#方法三：
我们也可以仿照RFPDupeFilter自定义去重规则，

from scrapy.dupefilter import RFPDupeFilter，看源码，仿照BaseDupeFilter

#步骤一：在项目目录下自定义去重文件cumstomdupefilter.py
'''
if hasattr("MyDupeFilter",from_settings):
     func = getattr("MyDupeFilter",from_settings)
        obj = func()
     else:
        return MyDupeFilter()
'''
class MyDupeFilter(object):
    def __init__(self):
        self.visited = set()

    @classmethod
    def from_settings(cls, settings):
        '''读取配置文件'''
        return cls()

    def request_seen(self, request):
        '''请求看过没有，这个才是去重规则该调用的方法'''
        if request.url in  self.visited:
            return True
        self.visited.add(request.url)

    def open(self):  # can return deferred
        '''打开的时候执行'''
        pass

    def close(self, reason):  # can return a deferred
        pass

    def log(self, request, spider):  # log that a request has been filtered
        '''日志记录'''
        pass

#步骤二：配置文件settings.py
# DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter'  #默认会去找这个类实现去重
#自定义去重规则
DUPEFILTER_CLASS = 'AMAZON.cumstomdupefilter.MyDupeFilter'

# 源码分析：
from scrapy.core.scheduler import Scheduler
见Scheduler下的enqueue_request方法：self.df.request_seen(request)

 

 

 

自定义扩展：

自定义扩展（与django的信号类似）
    1、django的信号是django是预留的扩展，信号一旦被触发，相应的功能就会执行
    2、scrapy自定义扩展的好处是可以在任意我们想要爬虫执行过程中添加功能，而其他组件中提供的功能只能在规定的位置执行

'''

engine_started = object()
engine_stopped = object()
spider_opened = object()
spider_idle = object()
spider_closed = object()
spider_error = object()
request_scheduled = object()
request_dropped = object()
response_received = object()
response_downloaded = object()
item_scraped = object()
item_dropped = object()
'''

from scrapy import signals


# 1、在与settings同级目录下新建一个文件，文件名可以为extentions.py,内容如下
class MyExtension(object):
    def __init__(self, value):
        self.value = value

    @classmethod
    def from_crawler(cls, crawler):
        val = crawler.settings.getint('MMMM')
        ext = cls(val)
        print('in the signals=================================')
        # crawler.signals.connect： 在scrapy中注册信号
        crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
        crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)

        return ext

    def spider_opened(self, spider):
        print('signals==========================open')

    def spider_closed(self, spider):
        print('signals==========================close')

# 配置生效
# EXTENSIONS = {
#     'sp1.extensions.MyExtension': 200,
# }

 

 

scrapy中间件：

定义在项目中的middlewares.py中，中间件的类名可以自定义，在配置文件中声明即可

爬虫中间件：

当需要对所有的爬虫进行统一的处理时，考虑使用爬虫中间件：

class SpiderMiddleware(object):

    def process_spider_input(self,response, spider):
        """
        下载完成，执行，然后交给parse处理
        :param response:  Downloader下载后返回的response，里面还包含封装了response.request(上一次发送的请求)
        :param spider: 
        :return: 
        """
        pass

    def process_spider_output(self,response, result, spider):
        """
        spider上次爬取处理完成，重新发起爬取请求，yield返回时调用
        :param response: 上一次爬取的结果
        :param result:    yield出去的对象，可能是Item或Request
        :param spider:
        :return: 必须返回包含 Request 或 Item 对象的可迭代对象(iterable)
        """
        return result

    def process_spider_exception(self,response, exception, spider):
        """
        异常调用
        :param response:
        :param exception:
        :param spider:
        :return: None,继续交给后续中间件处理异常；含 Response 或 Item 的可迭代对象(iterable)，交给调度器或pipeline
        """
        return None


    def process_start_requests(self,start_requests, spider):
        """
        爬虫启动时调用
        :param start_requests:    爬虫传递过来的start_urls，可迭代对象
        :param spider:
        :return: 包含 Request 对象的可迭代对象
        """
        return start_requests        # 如果返回的不是start_requests，将替换爬取的url

# 注册，可以写多个中间件，值小的优先级高：
SPIDER_MIDDLEWARES = {
    'sp1.middlewares.Sp1SpiderMiddleware': 543,
}

 

下载中间件（常用设置请求头：代理设置）：

从scheduler过来的请求：经过下载中间价，可做如下方法处理：

# 从scheduler过来的请求：经过下载中间价，可做如下方法处理：
class DownMiddleware1(object):
    def process_request(self, request, spider):
        """
        请求需要被下载时，经过所有下载器中间件的process_request调用
        :param request: 请求相关所有的数据（url、callback、headers等等），可以在请求头中加入代理;之后所有的请求都自动加上代理了
        :param spider: 
        :return:  
            返回None,什么都不做，继续执行后续中间件的request去下载，应用场景：代理的设置；注意：可以在process_request方法中设置代理，返回None，继续后面的中间件执行；
            返回Response对象，表示已经获取到结果了，停止process_request的执行（用于自定义下载），开始执行process_response ；      from scrapy.http import response
            返回Request对象（不太常用），停止中间件的执行，将Request重新放入调度器，等待下一次重新调度发起request请求        ；      from scrapy.http import Request
            返回raise IgnoreRequest异常（不常用），停止process_request的执行，开始执行process_exception
        """
        '''
        # 应用场景1：设置代理
        request.method = "POST"  # 把所有的请求方法都改为POST
        request.headers['kkk'] = 'vvvv'
        request.headers['proxy'] = to_bytes（'ip_port': '111.11.228.75:80'）  #设置代理 
        return None
        '''
        
        """
        # 应用场景2：自定义下载
        from scrapy.http import Response
        import requests
        v = request.get('http://www.baidu.com')
        data = Response(url='xxxxxxxx',body=v.content,request=request)
        return data                                                    # 返回了一个response对象，表示已经获取到结果了，停止process_request的执行，不会再去执行下
         """
        pass



    def process_response(self, request, response, spider):            # process_response执行的时候已经下载完了
        """
        spider处理完成，返回时调用，可用于响应头响应体的解析
        :param response:
        :param result:
        :param spider:
        :return: 
            返回Response 对象：转交给其他中间件process_response；
            返回Request 对象：停止中间件，又放回调度器中，request会被重新调度下载
            raise IgnoreRequest 异常：调用Request.errback
        """
        print('response1')
        '''
        场景1：返回数据统一编码；
        # from scrapy.http import Response
        # response.encoding = 'utf-8'        # 把所有返回都变成utf8编码,response.text得到的都是utf8编码类型
        场景2：
        根据业务需求重写response类
        '''
        return response

    def process_exception(self, request, exception, spider):
        """
        Scrapy calls process_exception() when a download handler or a process_request() (from a downloader middleware) raises an exception (including an IgnoreRequest exception)
        当下载处理器(download handler)或 process_request() (下载中间件)抛出异常如：超时异常、连接失败异常，scrapy会捕捉到异常，会自动调用process_exception
        :param response:
        :param exception:
        :param spider:
        :return: 
            None：继续交给后续中间件的process_exception处理异常；
            Response对象：停止后续process_exception方法，执行其他中间件的process_response方法，不会再调用process_exception
            Request对象：停止中间件，request将会被重新调用下载，得到异常之后重新发起请求，进行失败的request重试（如失败后再加一些参数、代理...）
        """
        return request        # 异常后重新发起请求

        
#DOWNLOADER_MIDDLEWARES = {
#    'sp3.middlewares.Sp3DownloaderMiddleware': 543,
#}
        

 

 

# middlewares.py中定义：
class RandomUserAgentMiddleware(object):
    def __init__(self, arg):
        self.user_agents = arg

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        val = crawler.settings.get('USER_AGENT')
        return cls(str(val))

    def process_request(self, request, spider):
        ret = re.sub('\n\s+', '', self.user_agents)
        result = ret.split(',')
        request.headers['User-Agent'] = random.choice(result)
        
        
        
        
# settings中添加：

USER_AGENT = """
    'Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)',
    'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/22.0.1216.0 Safari/537.2',
    'Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:15.0) Gecko/20100101 Firefox/15.0.1'
"""

# 测试验证：
class HttpbinSpider(scrapy.Spider):
    name = 'httpbin'
    allowed_domains = ['httpbin.org']
    start_urls = ['http://httpbin.org/get']

    def parse(self, response):
        self.logger.warn(response.text)

 

# scrapy设置动态随机的User-Agent

settings配置：
USER_AGENTS = [
    'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)'，
    'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Avant Browser)',
    ]


middlewares.py中配置：
from settings import USER_AGENTS

class RandomUserAgent(object):
    def process_request(self, request, spider):
        useragent = random.choice(USER_AGENTS)

        request.headers.setdefault("User-Agent", useragent)

import json
import logging
from scrapy import signals
import requests


class ProxyMiddleware():
    def __init__(self, proxy_url):
        self.logger = logging.getLogger(__name__)
        self.proxy_url = proxy_url
    
    def get_random_proxy(self):
        try:
            response = requests.get(self.proxy_url)
            if response.status_code == 200:
                proxy = response.text
                return proxy
        except requests.ConnectionError:
            return False
    
    def process_request(self, request, spider):
        if request.meta.get('retry_times'):
            proxy = self.get_random_proxy()
            if proxy:
                uri = 'https://{proxy}'.format(proxy=proxy)
                self.logger.debug('使用代理 ' + proxy)
                request.meta['proxy'] = uri

    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings
        return cls(
            proxy_url=settings.get('PROXY_URL')
        )


class CookiesMiddleware():
    def __init__(self, cookies_url):
        self.logger = logging.getLogger(__name__)
        self.cookies_url = cookies_url
    
    def get_random_cookies(self):
        try:
            response = requests.get(self.cookies_url)
            if response.status_code == 200:
                cookies = json.loads(response.text)
                return cookies
        except requests.ConnectionError:
            return False
    
    def process_request(self, request, spider):
        self.logger.debug('正在获取Cookies')
        cookies = self.get_random_cookies()
        if cookies:
            request.cookies = cookies
            self.logger.debug('使用Cookies ' + json.dumps(cookies))
    
    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings
        return cls(
            cookies_url=settings.get('COOKIES_URL')
        )

middlewares中：
class ZhihuuserProxyMiddleware(object):

    logger = logging.getLogger(__name__)

    def process_request(self, request, spider):
        self.logger.debug('Using Proxy------------------------')
        request.meta['proxy'] = '92.222.150.204:2028'
        
        
settings中：
DOWNLOADER_MIDDLEWARES = {
   'zhihuuser.middlewares.ZhihuuserProxyMiddleware': 543,
}
       

 

 

 

自定制命令:

• 在spiders同级创建任意目录，如：commands
• 在其中创建 crawlall.py 文件 （此处文件名就是自定义的命令）

from scrapy.commands import ScrapyCommand
    from scrapy.utils.project import get_project_settings


    class Command(ScrapyCommand):

        requires_project = True

        def syntax(self):
            return '[options]'

        def short_desc(self):
            return 'Runs all of the spiders'

        def run(self, args, opts):
            # 爬虫列表
            spider_list = self.crawler_process.spiders.list()        # 获取spiders目录下所有爬虫
            for name in spider_list:
                # 初始化爬虫
                self.crawler_process.crawl(name, **opts.__dict__)
            # 开始执行所有的爬虫
            self.crawler_process.start()

• 在settings.py 中添加配置 COMMANDS_MODULE = '项目名称.目录名称'
• 在项目目录执行命令：scrapy crawlall 

 

代理设置、https自定义证书、配置文件详解：

# -*- coding: utf-8 -*-

# Scrapy settings for step8_king project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     http://doc.scrapy.org/en/latest/topics/settings.html
#     http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
#     http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html

# 1. 爬虫名称
BOT_NAME = 'step8_king'

# 2. 爬虫应用路径
SPIDER_MODULES = ['step8_king.spiders']
NEWSPIDER_MODULE = 'step8_king.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
# 3. 客户端 user-agent请求头
# USER_AGENT = 'step8_king (+http://www.yourdomain.com)'

# Obey robots.txt rules
# 4. 禁止爬虫配置
# ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
# 5. 并发请求数
# CONCURRENT_REQUESTS = 4

# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
# 6. 延迟下载秒数
# DOWNLOAD_DELAY = 2


# The download delay setting will honor only one of:
# 7. 单域名访问并发数，并且延迟下次秒数也应用在每个域名
# CONCURRENT_REQUESTS_PER_DOMAIN = 2
# 单IP访问并发数，如果有值则忽略：CONCURRENT_REQUESTS_PER_DOMAIN，并且延迟下次秒数也应用在每个IP
# CONCURRENT_REQUESTS_PER_IP = 3
# 注意：设置上面两个参数会覆盖CONCURRENT_REQUESTS参数的值

# Disable cookies (enabled by default)
# 8. 是否支持cookie，cookiejar进行操作cookie
# COOKIES_ENABLED = True
# COOKIES_DEBUG = True        # 会在爬虫执行日志中打印出cookie信息

# Disable Telnet Console (enabled by default)
# 9. Telnet用于查看当前爬虫的信息，操作爬虫等...
#    使用telnet ip port ，然后通过命令操作    ； 用于监控爬虫
# TELNETCONSOLE_ENABLED = True
# TELNETCONSOLE_HOST = '127.0.0.1'
# TELNETCONSOLE_PORT = [6023,]


# 10. 对所有的请求，默认请求头，在spiders中定义的headers优先
# Override the default request headers:
# DEFAULT_REQUEST_HEADERS = {
#     'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#     'Accept-Language': 'en',
# }


# Configure item pipelines
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
# 11. 定义pipeline处理请求
# ITEM_PIPELINES = {
#    'step8_king.pipelines.JsonPipeline': 700,            # 值范围：[0-1000]
#    'step8_king.pipelines.FilePipeline': 500,
# }



# 12. 自定义扩展，基于信号进行调用
# Enable or disable extensions
# See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
# EXTENSIONS = {
#     # 'step8_king.extensions.MyExtension': 500,
# }


# 13. 爬虫允许的最大深度，可以通过meta查看当前深度；0表示无深度
# DEPTH_LIMIT = 3

# 14. 爬取时，0表示深度优先Lifo(默认)；1表示广度优先FiFo

# 后进先出，深度优先
# DEPTH_PRIORITY = 0
# SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleLifoDiskQueue'
# SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.LifoMemoryQueue'
# 先进先出，广度优先

# DEPTH_PRIORITY = 1
# SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue'
# SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'

# 15. 调度器队列
# SCHEDULER = 'scrapy.core.scheduler.Scheduler'
# from scrapy.core.scheduler import Scheduler


# 16. 访问URL去重
# DUPEFILTER_CLASS = 'step8_king.duplication.RepeatUrl'


# Enable and configure the AutoThrottle extension (disabled by default)
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html

"""
17. 自动限速算法
    from scrapy.contrib.throttle import AutoThrottle
    自动限速设置
    1. 获取最小延迟 DOWNLOAD_DELAY
    2. 获取最大延迟 AUTOTHROTTLE_MAX_DELAY
    3. 设置初始下载延迟 AUTOTHROTTLE_START_DELAY
    4. 当请求下载完成后，获取其"连接"时间 latency，即：请求连接到接受到响应头之间的时间
    5. 用于计算的... AUTOTHROTTLE_TARGET_CONCURRENCY
    target_delay = latency / self.target_concurrency
    new_delay = (slot.delay + target_delay) / 2.0 # 表示上一次的延迟时间
    new_delay = max(target_delay, new_delay)
    new_delay = min(max(self.mindelay, new_delay), self.maxdelay)
    slot.delay = new_delay
"""

# 开始自动限速
# AUTOTHROTTLE_ENABLED = True
# The initial download delay
# 初始下载延迟
# AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
# 最大下载延迟
# AUTOTHROTTLE_MAX_DELAY = 10
# The average number of requests Scrapy should be sending in parallel to each remote server
# 平均每秒并发数
# AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

# Enable showing throttling stats for every response received:
# 是否显示
# AUTOTHROTTLE_DEBUG = True

# Enable and configure HTTP caching (disabled by default)
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings


"""
18. 启用缓存                # 没有网络时、测试时可以使用
    目的用于将已经发送的请求或相应缓存下来，以便以后使用
    
    from scrapy.downloadermiddlewares.httpcache import HttpCacheMiddleware
    from scrapy.extensions.httpcache import DummyPolicy
    from scrapy.extensions.httpcache import FilesystemCacheStorage
"""
# 是否启用缓存策略
# HTTPCACHE_ENABLED = True

# 缓存策略：所有请求均缓存，下次在请求直接访问原来的缓存即可
# HTTPCACHE_POLICY = "scrapy.extensions.httpcache.DummyPolicy"
# 缓存策略：根据Http响应头：Cache-Control、Last-Modified 等进行缓存的策略
# HTTPCACHE_POLICY = "scrapy.extensions.httpcache.RFC2616Policy"

# 缓存超时时间
# HTTPCACHE_EXPIRATION_SECS = 0

# 缓存保存路径
# HTTPCACHE_DIR = 'httpcache'

# 缓存忽略的Http状态码
# HTTPCACHE_IGNORE_HTTP_CODES = []

# 缓存存储的插件
# HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'


"""
19. 代理，需要在环境变量中设置
    from scrapy.contrib.downloadermiddleware.httpproxy import HttpProxyMiddleware
    
    方式一：使用默认【局限：不能（不方便）随机取代理池中的代理】
        在spiders中的爬虫中设置
        import os
        os.environ
        {
            http_proxy:http://root:woshiniba@192.168.11.11:9999/
            https_proxy:http://192.168.11.11:9999/
        }
        import os
        # 设置的key必须带有_proxy后缀名
        os.environ['http_proxy'] = "http://root:woshiniba@192.168.11.11:9999/"
        os.environ['https_proxy'] = "http://192.168.11.11:9999/"
        os.environ['xx_proxy'] = "http://192.168.11.11:9999/"
        
    方式二：使用自定义下载中间件（参考默认HttpProxyMiddleware来写的），可以
    def to_bytes(text, encoding=None, errors='strict'):
        if isinstance(text, bytes):
            return text
        if not isinstance(text, six.string_types):
            raise TypeError('to_bytes must receive a unicode, str or bytes '
                            'object, got %s' % type(text).__name__)
        if encoding is None:
            encoding = 'utf-8'
        return text.encode(encoding, errors)
        
    class ProxyMiddleware(object):                        # 类名自定义，在配置文件中DOWNLOADER_MIDDLEWARES变量中写好就OK
        def process_request(self, request, spider):        # 重写下载中间件的process_request方法
            PROXIES = [
                {'ip_port': '111.11.228.75:80', 'user_pass': ''},
                {'ip_port': '120.198.243.22:80', 'user_pass': ''},
                {'ip_port': '111.8.60.9:8123', 'user_pass': ''},
                {'ip_port': '101.71.27.120:80', 'user_pass': ''},
                {'ip_port': '122.96.59.104:80', 'user_pass': ''},
                {'ip_port': '122.224.249.122:8088', 'user_pass': ''}, 
            ]
            import random  
            proxy = random.choice(PROXIES)
            if proxy['user_pass'] is not None:
                request.meta['proxy'] = to_bytes（"http://%s" % proxy['ip_port']）             # python3要转字节，to_bytes方法参考HttpProxyMiddleware中的to_bytes方法
                encoded_user_pass = base64.encodestring(to_bytes(proxy['user_pass']))
                request.headers['Proxy-Authorization'] = to_bytes('Basic ' + encoded_user_pass)    # 代理本质加这个Proxy-Authorization请求头
                print "**************ProxyMiddleware have pass************" + proxy['ip_port']
            else:
                print "**************ProxyMiddleware no pass************" + proxy['ip_port']
                request.meta['proxy'] = to_bytes("http://%s" % proxy['ip_port'])
    
    DOWNLOADER_MIDDLEWARES = {
       'step8_king.middlewares.ProxyMiddleware': 500,
    }
    
"""

"""
20. Https访问（本质通过下载中间件实现）
    Https访问时有两种情况：
    1. 要爬取网站使用的可信任证书(默认)
        DOWNLOADER_HTTPCLIENTFACTORY = "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory"
        DOWNLOADER_CLIENTCONTEXTFACTORY = "scrapy.core.downloader.contextfactory.ScrapyClientContextFactory"
        
    2. 要爬取网站使用的自定义证书，配置文件中加入如下配置：
        DOWNLOADER_HTTPCLIENTFACTORY = "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory"
        DOWNLOADER_CLIENTCONTEXTFACTORY = "step8_king.https.MySSLFactory"
        
        # https.py
        from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
        from twisted.internet.ssl import (optionsForClientTLS, CertificateOptions, PrivateCertificate)
        
        class MySSLFactory(ScrapyClientContextFactory):
            def getCertificateOptions(self):
                from OpenSSL import crypto
                v1 = crypto.load_privatekey(crypto.FILETYPE_PEM, open('/Users/wupeiqi/client.key.unsecure', mode='r').read())
                v2 = crypto.load_certificate(crypto.FILETYPE_PEM, open('/Users/wupeiqi/client.pem', mode='r').read())
                return CertificateOptions(
                    privateKey=v1,  # pKey对象
                    certificate=v2,  # X509对象
                    verify=False,
                    method=getattr(self, 'method', getattr(self, '_ssl_method', None))
                )
    其他：
        相关类
            scrapy.core.downloader.handlers.http.HttpDownloadHandler
            scrapy.core.downloader.webclient.ScrapyHTTPClientFactory
            scrapy.core.downloader.contextfactory.ScrapyClientContextFactory
        相关配置
            DOWNLOADER_HTTPCLIENTFACTORY
            DOWNLOADER_CLIENTCONTEXTFACTORY

"""



"""
21. 爬虫中间件
    class SpiderMiddleware(object):

        def process_spider_input(self,response, spider):
            '''
            下载完成，执行，然后交给parse处理
            :param response: 
            :param spider: 
            :return: 
            '''
            pass
    
        def process_spider_output(self,response, result, spider):
            '''
            spider处理完成，返回时调用
            :param response:
            :param result:
            :param spider:
            :return: 必须返回包含 Request 或 Item 对象的可迭代对象(iterable)
            '''
            return result
    
        def process_spider_exception(self,response, exception, spider):
            '''
            异常调用
            :param response:
            :param exception:
            :param spider:
            :return: None,继续交给后续中间件处理异常；含 Response 或 Item 的可迭代对象(iterable)，交给调度器或pipeline
            '''
            return None
    
    
        def process_start_requests(self,start_requests, spider):
            '''
            爬虫启动时调用
            :param start_requests:
            :param spider:
            :return: 包含 Request 对象的可迭代对象
            '''
            return start_requests
    
    内置爬虫中间件：
        'scrapy.contrib.spidermiddleware.httperror.HttpErrorMiddleware': 50,
        'scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware': 500,
        'scrapy.contrib.spidermiddleware.referer.RefererMiddleware': 700,
        'scrapy.contrib.spidermiddleware.urllength.UrlLengthMiddleware': 800,
        'scrapy.contrib.spidermiddleware.depth.DepthMiddleware': 900,

"""
# from scrapy.contrib.spidermiddleware.referer import RefererMiddleware
# Enable or disable spider middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
SPIDER_MIDDLEWARES = {
   # 'step8_king.middlewares.SpiderMiddleware': 543,
}


"""
22. 下载中间件
    class DownMiddleware1(object):
        def process_request(self, request, spider):
            '''
            请求需要被下载时，经过所有下载器中间件的process_request调用
            :param request:
            :param spider:
            :return:
                None,继续后续中间件去下载；
                Response对象，停止process_request的执行，开始执行process_response
                Request对象，停止中间件的执行，将Request重新调度器
                raise IgnoreRequest异常，停止process_request的执行，开始执行process_exception
            '''
            pass
    
    
    
        def process_response(self, request, response, spider):
            '''
            spider处理完成，返回时调用
            :param response:
            :param result:
            :param spider:
            :return:
                Response 对象：转交给其他中间件process_response
                Request 对象：停止中间件，request会被重新调度下载
                raise IgnoreRequest 异常：调用Request.errback
            '''
            print('response1')
            return response
    
        def process_exception(self, request, exception, spider):
            '''
            当下载处理器(download handler)或 process_request() (下载中间件)抛出异常
            :param response:
            :param exception:
            :param spider:
            :return:
                None：继续交给后续中间件处理异常；
                Response对象：停止后续process_exception方法
                Request对象：停止中间件，request将会被重新调用下载
            '''
            return None

    
    默认下载中间件
    {
        'scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware': 100,
        'scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware': 300,
        'scrapy.contrib.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware': 350,
        'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': 400,
        'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 500,
        'scrapy.contrib.downloadermiddleware.defaultheaders.DefaultHeadersMiddleware': 550,
        'scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware': 580,
        'scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware': 590,
        'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': 600,
        'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware': 700,
        'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 750,
        'scrapy.contrib.downloadermiddleware.chunked.ChunkedTransferMiddleware': 830,
        'scrapy.contrib.downloadermiddleware.stats.DownloaderStats': 850,
        'scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware': 900,
    }

"""
# from scrapy.contrib.downloadermiddleware.httpauth import HttpAuthMiddleware
# Enable or disable downloader middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
# DOWNLOADER_MIDDLEWARES = {
#    'step8_king.middlewares.DownMiddleware1': 100,
#    'step8_king.middlewares.DownMiddleware2': 500,
# }

 

 

TinyScrapy

#!/usr/bin/env python
# -*- coding:utf-8 -*-
from twisted.web.client import getPage, defer
from twisted.internet import reactor

# 1. 基本使用
"""
def all_done(arg):            # 所有爬虫执行完后，循环终止
    reactor.stop()

def callback(contents):        # 每一个爬虫获取结果后自动执行
    print(contents)


deferred_list = []

url_list = ['http://www.bing.com', 'http://www.baidu.com', ]
for url in url_list:
    deferred = getPage(bytes(url, encoding='utf8'))
    deferred.addCallback(callback)
    deferred_list.append(deferred)

dlist = defer.DeferredList(deferred_list)
dlist.addBoth(all_done)

reactor.run()  # 事件循环；监测哪个socket完成了
"""


# 2. 基于装饰器（一）
"""
def all_done(arg):
    reactor.stop()


def onedone(response):
    print(response)


@defer.inlineCallbacks
def task(url):
    deferred = getPage(bytes(url, encoding='utf8'))
    deferred.addCallback(onedone)
    yield deferred


deferred_list = []

url_list = ['http://www.bing.com', 'http://www.baidu.com', ]
for url in url_list:
    deferred = task(url)
    deferred_list.append(deferred)

dlist = defer.DeferredList(deferred_list)
dlist.addBoth(all_done)

reactor.run()
"""


# 3. 基于装饰器（二）
"""
def all_done(arg):
    reactor.stop()


def onedone(response):
    print(response)


@defer.inlineCallbacks
def task():
    deferred2 = getPage(bytes("http://www.baidu.com", encoding='utf8'))
    deferred2.addCallback(onedone)
    yield deferred2


    deferred1 = getPage(bytes("http://www.google.com", encoding='utf8'))
    deferred1.addCallback(onedone)
    yield deferred1


ret = task()
ret.addBoth(all_done)

reactor.run()
"""



# 4. 基于装饰器，永恒循环
"""
def all_done(arg):
    reactor.stop()


def onedone(response):
    print(response)


@defer.inlineCallbacks
def task():
    deferred2 = getPage(bytes("http://www.baidu.com", encoding='utf8'))
    deferred2.addCallback(onedone)
    yield deferred2

    stop_deferred = defer.Deferred()        # 创建了一个空的deferred对象，永远夯住；scrapy利用这一点保持一直运行，队列不断接收爬取请求，直到队列中没有请求，再调用callback停止scrapy运行
    # stop_deferred.callback()              # 执行回调函数run监测循环立即终止
    yield stop_deferred                    # 返回Deferred（）循环一直运行，不终止


ret = task()
ret.addBoth(all_done)

reactor.run()
"""


# 5. 基于装饰器，执行完毕后停止事件循环
"""
running_list = []
stop_deferred = None

def all_done(arg):
    reactor.stop()

def onedone(response, url):
    print(response)
    running_list.remove(url)
    
def check_empty(response):
    if not running_list:
        stop_deferred.callback('......')

@defer.inlineCallbacks
def task(url):
    deferred2 = getPage(bytes(url, encoding='utf8'))
    deferred2.addCallback(onedone, url)
    deferred2.addCallback(check_empty)
    yield deferred2

    stop_deferred = defer.Deferred()
    yield stop_deferred


ret = task("http://www.baidu.com")
ret.addBoth(all_done)

reactor.run()
"""


# reactor.callLater(0) # 结束当前Deferred，事件循环也会终止



# 6. 基于装饰器，执行完毕后停止事件循环
"""
import queue

running_list = []
stop_deferred = None
q = queue.Queue()


def all_done(arg):
    reactor.stop()

def onedone(response):
    print(response)

def check_empty(response):
    if not running_list:
        stop_deferred.callback('......')


def open_spider():
    url = q.get()
    deferred = getPage(bytes(url, encoding='utf8'))
    deferred.addCallback(onedone)
    deferred.addCallback(check_empty)


@defer.inlineCallbacks
def task(start_url):
    q.put(start_url)
    open_spider()

    global stop_deferred
    stop_deferred = defer.Deferred()
    yield stop_deferred

li = []
ret = task("http://www.baidu.com")
li.append(ret)

lid = defer.DeferredList(li)
lid.addBoth(all_done)

reactor.run()
"""

#!/usr/bin/env python
# -*- coding:utf-8 -*-
from twisted.web.client import getPage, defer
from twisted.internet import reactor
import queue


class Response(object):
    def __init__(self, body, request):
        self.body = body
        self.request = request
        self.url = request.url

    @property
    def text(self):
        return self.body.decode('utf-8')


class Request(object):
    def __init__(self, url, callback=None):
        self.url = url
        self.callback = callback


class Scheduler(object):
    def __init__(self, engine):
        self.q = queue.Queue()
        self.engine = engine

    def enqueue_request(self, request):
        self.q.put(request)

    def next_request(self):
        try:
            req = self.q.get(block=False)
        except Exception as e:
            req = None

        return req

    def size(self):
        return self.q.qsize()


class ExecutionEngine(object):
    def __init__(self):
        self._closewait = None
        self.running = True
        self.start_requests = None
        self.scheduler = Scheduler(self)

        self.inprogress = set()

    def check_empty(self, response):
        if not self.running:
            self._closewait.callback('......')

    def _next_request(self):
        while self.start_requests:
            try:
                request = next(self.start_requests)
            except StopIteration:
                self.start_requests = None
            else:
                self.scheduler.enqueue_request(request)

        while len(self.inprogress) < 5 and self.scheduler.size() > 0:  # 最大并发数为5
            # while循环不会夯住；
            request = self.scheduler.next_request()
            if not request:
                break

            self.inprogress.add(request)
            d = getPage(bytes(request.url, encoding='utf-8'))
            # 回调函数说明：
            # 如果getPage下载页面出错会执行d.addError；如果执行正确，会执行d.addCallback函数；addBoth不管结果怎么样都会执行
            d.addBoth(self._handle_downloader_output, request)                # 如果有数据回来，执行_handle_downloader_output
            d.addBoth(lambda x, req: self.inprogress.remove(req), request)
            d.addBoth(lambda x: self._next_request())
            
        if len(self.inprogress) == 0 and self.scheduler.size() == 0: #条件成立结束夯住scrapy的deferred()函数
            self._closewait.callback(None)

    def _handle_downloader_output(self, body, request):
        """
        获取内容，执行回调函数，并且把回调函数中的返回值获取，并添加到队列中
        :param response: 
        :param request: 
        :return: 
        """
        import types

        response = Response(body, request)
        func = request.callback or self.spider.parse
        gen = func(response)
        if isinstance(gen, types.GeneratorType):        # 如果是生成器（spiders yield过来的），继续放入队列执行；还可能是Item，直接去pipeline持久化
            for req in gen:
                self.scheduler.enqueue_request(req)

    @defer.inlineCallbacks
    def start(self):
        self._closewait = defer.Deferred()
        yield self._closewait

    @defer.inlineCallbacks
    def open_spider(self, spider, start_requests):
        self.start_requests = start_requests
        self.spider = spider
        yield None
        reactor.callLater(0, self._next_request)


class Crawler(object):
    def __init__(self, spidercls):
        self.spidercls = spidercls

        self.spider = None
        self.engine = None

    @defer.inlineCallbacks
    def crawl(self):
        self.engine = ExecutionEngine()        # 创建引擎
        self.spider = self.spidercls()        # 创建spider对象，spidercls是爬虫对象列表
        start_requests = iter(self.spider.start_requests())        # 拿到start_urls执行引擎
        yield self.engine.open_spider(self.spider, start_requests)
        yield self.engine.start()                # 夯住，让scrapy保持运行
        

class CrawlerProcess(object):
    def __init__(self):
        self._active = set()
        self.crawlers = set()

    def crawl(self, spidercls, *args, **kwargs):
        crawler = Crawler(spidercls)
        self.crawlers.add(crawler)
        
        d = crawler.crawl(*args, **kwargs)
        self._active.add(d)
        return d

    def start(self):
        dl = defer.DeferredList(self._active)
        dl.addBoth(self._stop_reactor)
        reactor.run()

    def _stop_reactor(self, _=None):
        reactor.stop()


class Spider(object):
    def start_requests(self):
        for url in self.start_urls:
            yield Request(url)


class ChoutiSpider(Spider):
    name = "chouti"
    start_urls = [
        'http://dig.chouti.com/',
    ]

    def parse(self, response):
        print(response.text)


class CnblogsSpider(Spider):
    name = "cnblogs"
    start_urls = [
        'http://www.cnblogs.com/',
    ]

    def parse(self, response):
        print(response.text)


if __name__ == '__main__':

    spider_cls_list = [ChoutiSpider, CnblogsSpider]

    crawler_process = CrawlerProcess()
    for spider_cls in spider_cls_list:
        crawler_process.crawl(spider_cls)

    crawler_process.start()