利用 scrapy-splash 对京东进行模拟点击并进行数据爬取

本人是第一次写博客，有写得不好的地方欢迎值出来，大家一起进步！

scrapy-splash的介绍

scrapy-splash模块主要使用了Splash. 所谓的Splash, 就是一个Javascript渲染服务。它是一个实现了HTTP API的轻量级浏览器，Splash是用Python实现的，同时使用Twisted和QT。Twisted（QT）用来让服务具有异步处理能力，以发挥webkit的并发能力。Splash的特点如下：

• 并行处理多个网页
• 得到HTML结果以及（或者）渲染成图片
• 关掉加载图片或使用 Adblock Plus规则使得渲染速度更快
• 使用JavaScript处理网页内容
• 使用Lua脚本
• 能在Splash-Jupyter Notebooks中开发Splash Lua scripts
• 能够获得具体的HAR格式的渲染信息

参考文档：https://www.cnblogs.com/jclian91/p/8590617.html

准备配置

• scrapy框架
• splash安装，windows用户通过虚拟机安装docker,Linux直接安装docker

页面分析

首先进入https://search.jd.com/ 网站搜索想要的书籍， 这里以 python3.7 书籍为例子。

 

点击搜索后发现京东是通过 js 来加载书籍数据的， 通过下来鼠标可以发现加载了更多的书籍数据（数据也可以通过京东的api来获取）

 

首先是模拟搜索，通过检查可得：

 

然后是模拟下拉，这里选择页面底部的这个元素作为模拟元素：

 

开始爬取

模拟点击的lua脚本并获取页数：

function main(splash, args)
  splash.images_enabled = false
  splash:set_user_agent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36')
  assert(splash:go(args.url))
  splash:wait(0.5)
  local input = splash:select("#keyword")
  input:send_text('python3.7')
  splash:wait(0.5)
  local form = splash:select('.input_submit')
  form:click()
  splash:wait(2)
  splash:runjs("document.getElementsByClassName('bottom-search')[0].scrollIntoView(true)")
  splash:wait(6)
  return splash:html()
end

 

同上有模拟下拉的代码：

function main(splash, args)
  splash.images_enabled = false
  splash:set_user_agent('Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36')
  assert(splash:go(args.url))
  splash:wait(2)
  splash:runjs("document.getElementsByClassName('bottom-search')[0].scrollIntoView(true)")
  splash:wait(6)
  return splash:html()
end

 

选择你想要获取的元素，通过检查获得。附上源码：

 

 

# -*- coding: utf-8 -*-
import scrapy
from scrapy import Request
from scrapy_splash import SplashRequest
from ..items import JdsplashItem



lua_script = '''
function main(splash, args)
  splash.images_enabled = false
  splash:set_user_agent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36')
  assert(splash:go(args.url))
  splash:wait(0.5)
  local input = splash:select("#keyword")
  input:send_text('python3.7')
  splash:wait(0.5)
  local form = splash:select('.input_submit')
  form:click()
  splash:wait(2)
  splash:runjs("document.getElementsByClassName('bottom-search')[0].scrollIntoView(true)")
  splash:wait(6)
  return splash:html()
end
'''

lua_script2 = '''
function main(splash, args)
  splash.images_enabled = false
  splash:set_user_agent('Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36')
  assert(splash:go(args.url))
  splash:wait(2)
  splash:runjs("document.getElementsByClassName('bottom-search')[0].scrollIntoView(true)")
  splash:wait(6)
  return splash:html()
end
'''

class JdBookSpider(scrapy.Spider):
    name = 'jd'
    allowed_domains = ['search.jd.com']
    start_urls = ['https://search.jd.com']

    def start_requests(self):
        #进入搜索页进行搜索
        for each in self.start_urls:
            yield SplashRequest(each,callback=self.parse,endpoint='execute',
                args={'lua_source': lua_script})

    def parse(self, response):
        item = JdsplashItem()
        price = response.css('div.gl-i-wrap div.p-price i::text').getall()
        page_num = response.xpath("//span[@class= 'p-num']/a[last()-1]/text()").get()
        #这里使用了 xpath 函数 fn:string(arg):返回参数的字符串值。参数可以是数字、逻辑值或节点集。
        #可能这就是 xpath 比 css 更精致的地方吧
        name = response.css('div.gl-i-wrap div.p-name').xpath('string(.//em)').getall()
        #comment = response.css('div.gl-i-wrap div.p-commit').xpath('string(.//strong)').getall()
        comment = response.css('div.gl-i-wrap div.p-commit strong a::text').getall()
        publishstore = response.css('div.gl-i-wrap div.p-shopnum a::attr(title)').getall()
        href = [response.urljoin(i) for i in response.css('div.gl-i-wrap div.p-img a::attr(href)').getall()]
        for each in zip(name, price, comment, publishstore,href):
            item['name'] = each[0]
            item['price'] = each[1]
            item['comment'] = each[2]
            item['p_store'] = each[3]
            item['href'] = each[4]
            yield item
        #这里从第二页开始
        url = 'https://search.jd.com/Search?keyword=python3.7&enc=utf-8&qrst=1&rt=1&stop=1&vt=2&page=%d&s=%d&click=0'
        for each_page in range(1,int(page_num)):
            yield SplashRequest(url%(each_page*2+1,each_page*60),callback=self.s_parse,endpoint='execute',
                args={'lua_source': lua_script2})

    def s_parse(self, response):
        item = JdsplashItem()
        price = response.css('div.gl-i-wrap div.p-price i::text').getall()
        name = response.css('div.gl-i-wrap div.p-name').xpath('string(.//em)').getall()
        comment = response.css('div.gl-i-wrap div.p-commit strong a::text').getall()
        publishstore = response.css('div.gl-i-wrap div.p-shopnum a::attr(title)').getall()
        href = [response.urljoin(i) for i in response.css('div.gl-i-wrap div.p-img a::attr(href)').getall()]
        for each in zip(name, price, comment, publishstore, href):
            item['name'] = each[0]
            item['price'] = each[1]
            item['comment'] = each[2]
            item['p_store'] = each[3]
            item['href'] = each[4]
            yield item

 

各个文件的配置：

items.py :

import scrapy


class JdsplashItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    name = scrapy.Field()
    price = scrapy.Field()
    p_store = scrapy.Field()
    comment = scrapy.Field()
    href = scrapy.Field()
    pass

settings.py:

import scrapy_splash
# Splash服务器地址
SPLASH_URL = 'http://192.168.99.100:8050'
# 开启Splash的两个下载中间件并调整HttpCompressionMiddleware的次序
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

 

最后运行代码，可以看到书籍数据已经被爬取了：