初识scrapy框架（十）------ 爬取手机APP的图片

　　该文章以爬取手机斗鱼APP为例，我们希望爬取关键字”颜值“里面的主播大图，点击获取：前期手机配置与fiddler配置。

配置好fiddler和手机之后，打开斗鱼APP，用fiddler抓包，这里的数据返回的都是JSON数据，所以我们可以直接用json提取。

首先找到链接，这里作者抓到的链接使用是有问题的，真实科研的链接是：https://capi.douyucdn.cn*******&offset=0  .这

里不方便公布别人的资源链接，请自己多抓，自行寻找。链接效果图如下：

 

根据这个链接，使用scrapy爬取数据：

创建爬虫项目：scrapy startproject douyu

创建爬虫文件：scrapy genspider yanzhi  douyucdn.cn

明确爬虫目标：

import scrapy


class DouyuItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    # 主播名
    nick_name = scrapy.Field()
    # 房间号
    room_id = scrapy.Field()
    # 作者所在城市
    anchor_city = scrapy.Field()
    # 图片链接
    image_link = scrapy.Field()
    # 文件储存的地址
    image_path = scrapy.Field()

    source = scrapy.Field()
    utc_time = scrapy.Field()

编写爬虫文件：

import scrapy
import json
from douyu.items import DouyuItem


class YanzhiSpider(scrapy.Spider):
    name = 'yanzhi'
    allowed_domains = ['douyucdn.cn']
    offset = 0
    base_url = 'https://capi.douyucdn.cn**************&offset='
    start_urls = [base_url + str(offset)]

    def parse(self, response):
        node_list = json.loads(response.body.decode())["data"]

        if not node_list:
            return

        for node in node_list:
            item = DouyuItem()
            item["nick_name"] = node["nickname"]
            item["room_id"] = node["room_id"]
            item["anchor_city"] = node["anchor_city"]
            item["image_link"] = node["vertical_src"]
            yield item
        self.offset += 20
        yield scrapy.Request(url=self.base_url+str(self.offset), callback=self.parse)

编写管道文件：

from scrapy.pipelines.images import ImagesPipeline
from douyu.settings import IMAGES_STORE
from datetime import datetime
import scrapy
import os


class ImageSource(object):
    def process_item(self, item, spider):
        item["source"] = spider.name
        item["utc_time"] = str(datetime.utcnow())
        print("**"*20)
        return item


class DouyuImagesPipeline(ImagesPipeline):

    # 发送图片链接请求
    def get_media_requests(self, item, info):
        # 获取item数据的图片链接
        image_link = item["image_link"]
        print(image_link)
        # 发送图片请求，响应默认会保存在指定路径下
        yield scrapy.Request(url=image_link)

    def item_completed(self, results, item, info):
        # 每个result是一个图片信息，去除图片原来的路径
        image_path = [x["path"] for ok, x in results if ok]
        print(results)

        # 先保存当前图片的路径
        old_name = IMAGES_STORE + "/" + image_path[0]
        # 更改当前图片的路径名字
        new_name = IMAGES_STORE + "/" + item["nick_name"] + ".jpg"
        item["image_path"] = new_name
        try:
            # 修改为新的路径名，文件名
            os.rename(old_name, new_name)
        except Exception as e:
            print("[INFO]:图片已经修改\n", e)
        return item

编写下载中间件：

import random
from douyu.settings import USER_AGENTS as UA


class UserAgentMiddleware(object):

    """
        给每一次请求随机赋值一个User_Agent
    """

    def process_request(self, request, spider):
        user_agent = random.choice(UA)
        request.headers['User_Agent'] = user_agent

配置管道和下载中间件：

IMAGES_STORE = '/home/dan/data/images'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENTS = [
            "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",
            "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",
            "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",
            "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",
            "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",
            "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",
            "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",
            "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",
            "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3192.0 Safari/537.36Name"
            ]



# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
   'douyu.middlewares.UserAgentMiddleware': 543,
}


# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
    'douyu.pipelines.ImageSource': 100,
    'douyu.pipelines.DouyuImagesPipeline': 200,
}

这里的有些配置在前面已经配置好了，只是在这里同一书写。

为了方便执行与数据管理，我们创建一个py文件执行爬虫文件，并删除冗余的文件夹full

import os

print("开始执行爬虫程序")
os.system("scrapy crawl yanzhi")
print("删除多余的文件")
os.rmdir("/home/dan/data/images/full")

 

运行上面的代码，下面展示部分结果和数据：

文件夹下保存的数据：

上面的代码很简单，所以没有进行说明，如果没有基础还是先看看基础吧。

有木有很心动，代码已给，燥起来吧骚年！！！