8_1 scrapy入门实战（爬取糗事百科段子）

开始参考https://www.cnblogs.com/sruzzg/p/13060159.html

一、创建scrapy爬虫工程demo

scrapy startproject demo

快捷创建了一个demo新工程

 

二、在工程中生成一个scrapy爬虫qiushibaike

　　1：进入工程 cd demo

　　2：生成爬虫　　scrapy 　　genspider 　　爬虫名(和工程名不能相同) 　　域名

　　scrapy genspider qiushibaike  qiushibaike.com

 

三、修改setting文件

　　1：更改ROBOTSTXT_OBEY = False  设置为False

　　2：增加头文件

DEFAULT_REQUEST_HEADERS = {
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'en',
  'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
}

 

四、修改spiders文件下的qiushibaike.py文件

　　在这是里有3个重要属性：name、start_urls、方法parse()

其中def parse(self, response):中的response就是spiders发送给Engine再到Scheduler又返回Engine，再到Downloader，返回Engine，再到spiders的对url爬取的结果，所以很多操作都在parse()下进行。更改如下：

class QiushibaikeSpider(scrapy.Spider):
    name = 'qiushibaike'
    allowed_domains = ['qiushibaike.com']
    start_urls = ['https://www.qiushibaike.com/8hr/page/1/']

    def parse(self, response):
        print('='*40)
        print(type(response))
        print('='*40)

 

五、运行

　　1：命令行中运行

　　scrapy  crawl　qiushibaike

　　

　　2：pycharm中运行（调试更方便）

　　　　 setp1：在项目根目录新建python文件start

　　　　 step2：把命令行命令写入start文件中如下：

from scrapy import cmdline

# cmdline.execute(["scrapy", "crawl", "qiushibaike"])
cmdline.execute("scrapy crawl qiushibaike".split())

 

六、对糗事百科网站一个页面的爬取

# -*- coding: utf-8 -*-
import scrapy
from scrapy.http.response.html import HtmlResponse
from scrapy.selector.unified import SelectorList


class QiushibaikeSpider(scrapy.Spider):
    name = 'qiushibaike'
    allowed_domains = ['qiushibaike.com']
    start_urls = ['https://www.qiushibaike.com/text/page/2/']

    def parse(self, response):
        print("=" * 40)
        # SelectorList
        duanzidivs = response.xpath("//div[@class='col1 old-style-col1']/div")
        for duanzidiv in duanzidivs:
            # Selector
            author = duanzidiv.xpath(".//h2/text()").get().strip()
            print(author)
            content = duanzidiv.xpath(".//div[@class='content']//text()").getall()  # 返回列表
            content = "".join(content).strip()
            print(content)
        print("=" * 40)

附：

3 from scrapy.http.response.html import HtmlResponse
4 from scrapy.selector.unified import SelectorList
以上两行代码是用于查看response对应类HtmlResponse及SelectorList类的原码，可以查看里面的方法及属性


七、使用pipelines保存数据

　　1、更改QiushibaikeSpider()　

# -*- coding: utf-8 -*-
import scrapy

class QiushibaikeSpider(scrapy.Spider):
    name = 'qiushibaike'
    allowed_domains = ['qiushibaike.com']
    start_urls = ['https://www.qiushibaike.com/text/page/2/']

    def parse(self, response):
        # SelectorList
        duanzidivs = response.xpath("//div[@class='col1 old-style-col1']/div")
        for duanzidiv in duanzidivs:
            # Selector
            author = duanzidiv.xpath(".//h2/text()").get().strip()
            #print(author)
            content = duanzidiv.xpath(".//div[@class='content']//text()").getall()  # 返回列表
            content = "".join(content).strip()
            #print(content)
            duanzi = {"author":author, "conent":content}
            yield duanzi

　　2、pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
import json

class QiushibaikePipeline:
    def __init__(self):
        self.fp = open("duanzi.json", 'w', encoding='utf-8')

    def open_spider(self, spider):
        # 初始化代码写在这里也是一样的
        #self.fp = open("duanzi.json", 'w', encoding='utf-8')
        print('爬虫开始了...')

    def process_item(self, item, spider):
        # 数据的存处理
        item_json = json.dumps(item, ensure_ascii=False)　　　　# ensure_ascii=false显示中文
        self.fp.write(item_json+'\n')
        return item

    def close_spider(self, spider):
        self.fp.close()
        print('爬虫结束了...')

3、setting.py中设置pipelines

ITEM_PIPELINES = {
　　# 300表示优先级，越小越高
   'qiushibaike.pipelines.QiushibaikePipeline': 300,
}

 

八、优化存储（使用item代替字典，只是相对优化更好的见九）

　　1、修改items.py

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class QiushibaikeItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    author = scrapy.Field()
    content = scrapy.Field()

　　2、把类QiushibaikeItem在Qiushibaike用上

# -*- coding: utf-8 -*-
import scrapy
from scrapy.http.response.html import HtmlResponse
from scrapy.selector.unified import SelectorList
from qiushibaike.items import QiushibaikeItem


class DemoSpider(scrapy.Spider):
    name = 'qiushibaike'
    allowed_domains = ['qiushibaike.com']
    start_urls = ['https://www.qiushibaike.com/text/page/2/']

    def parse(self, response):
        # SelectorList
        duanzidivs = response.xpath("//div[@class='col1 old-style-col1']/div")
        for duanzidiv in duanzidivs:
            # Selector
            author = duanzidiv.xpath(".//h2/text()").get().strip()
            #print(author)
            content = duanzidiv.xpath(".//div[@class='content']//text()").getall()  # 返回列表
            content = "".join(content).strip()
            #print(content)
            #duanzi = {"author":author, "conent":content}
            item = QiushibaikeItem(author=author, content=content)
            yield item

3、更改pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
import json

class QiushibaikePipeline:
    def __init__(self):
        self.fp = open("duanzi.json", 'w', encoding='utf-8')

    def open_spider(self, spider):
        # 初始化代码写在这里也是一样的
        #self.fp = open("duanzi.json", 'w', encoding='utf-8')
        print('爬虫开始了...')

    def process_item(self, item, spider):
        # 数据的存处理
        item_json = json.dumps(dict(item), ensure_ascii=False)
        self.fp.write(item_json+'\n')
        return item

    def close_spider(self, spider):
        self.fp.close()
        print('爬虫结束了...')

 

 九、数据的存储方式

　　1、方法一：（都是存储在一个列表中，对数据不是非常大的情况可用）

　　

 1 from scrapy.exporters import JsonItemExporter

class QiushibaikePipeline:
    def __init__(self):
        self.fp = open("duanzi2.json", 'wb')
 6         self.exporter = JsonItemExporter(self.fp, ensure_ascii=False, encoding='utf-8')
 7         self.exporter.start_exporting()
        
    def open_spider(self, spider):
        # 初始化代码写在这里也是一样的
        print('爬虫开始了...')

    def process_item(self, item, spider):
        # 数据的存处理
        self.exporter.export_item(item)
        return item

    def close_spider(self, spider):
        self.exporter.finish_exporting()
        self.fp.close()
        print('爬虫结束了...')

　　2、方法二：（可处理大量数据存储为json格式，推荐使用）

 1 from scrapy.exporters import JsonLinesItemExporter
 2 
 3 class QiushibaikePipeline:
 4     def __init__(self):
 5         self.fp = open("duanzi3.json", 'wb')
 6         self.exporter = JsonLinesItemExporter(self.fp, ensure_ascii=False, encoding='utf-8')
 7 
 8     def open_spider(self, spider):
 9         # 初始化代码写在这里也是一样的
10         print('爬虫开始了...')
11 
12     def process_item(self, item, spider):
13         # 数据的存处理
14         self.exporter.export_item(item)
15         return item
16 
17     def close_spider(self, spider):
18         self.fp.close()
19         print('爬虫结束了...')

 

十、爬取多个页面

　　1、在Qiushibaike中对下一个页面的获取，并且使用回调函数再次爬取

# -*- coding: utf-8 -*-
import scrapy
from scrapy.http.response.html import HtmlResponse
from scrapy.selector.unified import SelectorList
from qiushibaike.items import QiushibaikeItem


class QiushibaikeSpider(scrapy.Spider):
    name = 'qiushibaike'
    allowed_domains = ['qiushibaike.com']
    start_urls = ['https://www.qiushibaike.com/text/page/2/']
    base_domain = "https://www.qiushibaike.com"

    def parse(self, response):
        # SelectorList
        duanzidivs = response.xpath("//div[@class='col1 old-style-col1']/div")
        for duanzidiv in duanzidivs:
            # Selector
            author = duanzidiv.xpath(".//h2/text()").get().strip()
            #print(author)
            content = duanzidiv.xpath(".//div[@class='content']//text()").getall()  # 返回列表
            content = "".join(content).strip()
            #print(content)
            #duanzi = {"author":author, "conent":content}
            item = QiushibaikeItem(author=author, content=content)
            yield item
        # 获取下一页网址
28         next_url = response.xpath("//ul[@class='pagination']/li[last()]/a/@href").get()
29         #print(next_url)
30         if not next_url:
31             # 最后一页了就为空
32             return
33         else:
34             next_url = self.base_domain + next_url
35             yield scrapy.Request(next_url, callback=self.parse)

　　2、对下载数据在setting中设置下载延时（太快对服务器不好）

DOWNLOAD_DELAY = 3

 

附：本笔记在写的时候用了两台电脑，不同电脑项目名称不同。可能在代码中demo与qiushibaike名称有点混（改过不知道有没有全改过来）