移动端爬虫基础

工具的安装及配置
练习

工具的安装及配置

1.抓包工具Fidder

安装完Fidder，要让Fidder作为手机APP的代理服务器，还需要进行相关的配置

证书配置：
Tools=》Options=》HTTPS，勾选Decrypt HTTPS traffic，表示解密HTTPS流量。勾选后会提示你安装根证书
Tools=》Options=》HTTPS=》Actions=》Open Windows Certificate Manager进行根证书的查看
信任证书的设置：Tools=》Options=》HTTPS=》Actions=》Trust Root Certificate
勾选其他如下选项
勾选允许远程计算机连接
关闭Fidder作为系统代理选项

2.夜神模拟器

安装完夜神模拟器进行相关的配置

选择32位的Android5版本的手机，启动手机，设置网络代理
访问http://ip:port进行根证书的安装。比如说Fidder开放的默认监听端口为8888，安装夜神模拟器的计算机网络地址为10.89.165.170，则访问地址http://10.89.165.170:8888/进行证书的下载
安装软件：将预先下载好的apk安装包拖动到模拟器中，进行安装
启用USB调试模式

3.Android SDK

新建环境变量ANDROID_HOME，其值为安装Android SDK的目录，例如E:\developTools\tools\AndroidSDK
在path变量下增加值%ANDROID_HOME%\platform-tools
检验环境变量是否配置成功：打开CMD窗口，输入adb version，查看adb的版本信息
问题：已经在夜神模拟器中启动了一个手机设备，但是使用adb devices查看没有对应的设备信息。解决如下：
1. 分别查看Android SDK中adb工具的版本以及安装的夜神模拟器中adb工具的版本
```
# 夜神模拟器
E:\developTools\spyder_tools\Nox\bin>adb version
Android Debug Bridge version 1.0.36
Revision 0e9850346394-android

# Android SDK
C:\Users\root>adb version
Android Debug Bridge version 1.0.41
Version 34.0.5-10900879
Installed as E:\developTools\tools\AndroidSDK\platform-tools\adb.exe
Running on Windows 10.0.22621
```
1. 如果两者不一致，则将复制Android SDK中adb工具，替换夜神模拟器中adb工具；复制Android SDK中adb工具，重命名为nox_adb.exe替换夜神模拟器中nox_adb.exe

4.Appium Server

安装完Appium进行相关的配置，这里安装的是版本是Appium-Server-GUI-windows-1.22.3-4

5.Appium inspector

这个工具方便定位HTML文档中的元素

下载地址：https://github.com/appium/appium-inspector/releases，这个工具可以作为Appium client。这里安装的版本是2023.12.2

配置如下

deviceName：通过adb devices命令获取
platformName：平台名称，比如说Android或者IOS
platformVersion：进入手机设备进行查看

appPackage/appActivity：

打开CMD窗口运行adb shell进入shell命令行
运行logcat | grep cmp=
打开手机设备，进入想要打开的应用程序。然后观察CMD窗口输出的信息

C:\Users\root>adb shell
root@shamu:/ # logcat | grep cmp=
I/ActivityManager( 1737): START u0 {act=android.intent.action.MAIN cat=[android.intent.category.HOME] flg=0x10000000 cmp=com.android.launcher3/.launcher3.Launcher} from uid 0 on display 0
W/ActivityManager( 1737): Unable to start service Intent { cmp=com.android.launcher3/.service.MonitorService } U=0: not found
I/ActivityManager( 1737): START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 cmp=com.android.settings/.Settings bnds=[34,535][234,695] (has extras)} from uid 1000 on display 0
I/ActivityManager( 1737): START u0 {act=android.intent.action.MAIN cmp=com.android.settings/.SubSettings (has extras)} from uid 1000 on display 0
I/ActivityManager( 1737): START u0 {act=android.intent.action.MAIN cmp=com.android.settings/.SubSettings (has extras)} from uid 1000 on display 0
I/ActivityManager( 1737): START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 cmp=com.android.settings/.Settings bnds=[34,535][234,695] (has extras)} from uid 1000 on display 0
I/ActivityManager( 1737): START u0 {act=android.intent.action.MAIN cmp=com.android.settings/.SubSettings (has extras)} from uid 1000 on display 0
I/ActivityManager( 1737): START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 cmp=com.android.browser/.BrowserActivity bnds=[18,553][192,812] (has extras)} from uid 1000 on display 0
W/ActivityManager( 1737): Unable to start service Intent { cmp=com.android.launcher3/.service.MonitorService } U=0: not found
I/ActivityManager( 1737): START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10000000 cmp=io.appium.settings/.Settings} from uid 0 on display 0
I/ActivityManager( 1737): START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 cmp=com.android.browser/.BrowserActivity} from uid 0 on display 0

最终的配置如下所示

{
  "appium:deviceName": "127.0.0.1:62025",
  "platformName": "Android",
  "appium:platformVersion": "5.1.1",
  "appium:appPackage": "com.android.browser",
  "appium:appActivity": ".BrowserActivity"
}

练习

安装：pip install Appium-Python-Client，版本为3.1.1

简单示例：爬取TOP500榜单音乐名称

根据仿真手机设备的版本下载相应的APK文件，拖动到仿真手机设备安装
示例脚本如下：

import time

from appium import webdriver
from appium.options.common import AppiumOptions
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


SERVER = 'http://localhost:4723/wd/hub'

options = AppiumOptions()
options.set_capability("platformName", "Android")
options.set_capability("appium:deviceName", "127.0.0.1:62025")
options.set_capability("appium:platformVersion", "5.1.1")
options.set_capability("appium:appPackage", "com.kugou.android")
options.set_capability("appium:appActivity", ".app.splash.SplashActivity")

driver = webdriver.Remote(command_executor=SERVER, options=options)

time.sleep(20)

# 获取模拟器的分辨率
width = driver.get_window_size()['width']
height = driver.get_window_size()['height']
start_x = end_x = int(width * 0.5)
#  手指滑动距离
start_y = int(height * 0.75)
end_y = int(height * 0.25)

# 初始化等待对象
wait = WebDriverWait(driver, 8)

# 同意协议信息
try:
    button_ok_xpath = '//android.widget.Button[@resource-id="com.kugou.android:id/a821"]'
    wait.until(EC.presence_of_element_located((By.XPATH, button_ok_xpath)))
    button_ok = driver.find_element(By.XPATH, button_ok_xpath)
    button_ok.click()
except Exception as e:
    pass

time.sleep(15)

# 点击跳过
try:
    button_skip_xpath = '//android.widget.TextView[@content-desc="跳过"]'
    wait.until(EC.presence_of_element_located((By.XPATH, button_skip_xpath)))
    button_skip = driver.find_element(By.XPATH, button_skip_xpath)
    button_skip.click()
except Exception as E:
    pass

# 点击取消注册
try:
    button_cancel_id = 'com.kugou.android:id/a0r'
    wait.until(EC.presence_of_element_located((By.ID, button_cancel_id)))
    button_cancel = driver.find_element(By.ID, button_cancel_id)
    button_cancel.click()
except Exception as e:
    pass

# 点击知道了
try:
    button_know_xpath = '//android.widget.Button[@resource-id="com.kugou.android:id/a6fx"]'
    wait.until(EC.presence_of_element_located((By.XPATH, button_know_xpath)))
    button_know = driver.find_element(By.XPATH, button_know_xpath)
    button_know.click()
except Exception as e:
    pass

# 点击排行榜
try:
    rank_list_xpath = '(//android.widget.ImageView[@resource-id="com.kugou.android:id/a64z"])[1]'
    wait.until(EC.presence_of_element_located((By.XPATH, rank_list_xpath)))
    rank_list = driver.find_element(By.XPATH, rank_list_xpath)
    rank_list.click()
except Exception as e:
    pass

# 点击TOP500
try:
    top_500_xpath = '//android.view.View[@resource-id="app-content"]/android.view.View[4]'
    wait.until(EC.presence_of_element_located((By.XPATH, top_500_xpath)))
    top_500 = driver.find_element(By.XPATH, top_500_xpath)
    top_500.click()
except Exception as e:
    pass


music_name_lst = []
flag = False
song_name_xpath = '//android.widget.TextView[@resource-id="com.kugou.android:id/b2w"]'

while not flag:
    name_lst = [ele.text for ele in driver.find_elements(By.XPATH, song_name_xpath)]
    for m in name_lst:
        if m not in music_name_lst:
            music_name_lst.append(m)
            print(m)
    time.sleep(0.5)
    driver.swipe(start_x, start_y, end_x, end_y)
    end_xpath = '//android.widget.TextView[@resource-id="com.kugou.android:id/dr5"]'
    try:
        flag = driver.find_element(By.XPATH, end_xpath)
    except Exception as e:
        flag = False

name_lst = [ele.text for ele in driver.find_elements(By.XPATH, song_name_xpath)]
for m in name_lst:
    if m not in music_name_lst:
        music_name_lst.append(m)
        print(m)

driver.close()

posted on 2024-01-25 14:52 scrutiny-span 阅读(52) 评论(0) 编辑收藏举报

刷新页面返回顶部

导航