空气质量检测平台js爬虫逆向分析

空气质量检测平台JS爬虫逆向分析

本文将展示如何使用Python构建一个爬虫，抓取空气质量检测平台的数据，并对其进行逆向分析。

1. 背景介绍

我们需要抓取空气质量检测平台提供的实时空气质量数据。在此过程中，我们遇到了一个常见的问题：请求的数据是经过加密的，需要我们对请求和响应的JS加密逻辑进行逆向分析。

2. 技术栈

Python 3.x
requests库：用于发送HTTP请求
execjs库：用于执行JavaScript代码
json库：用于解析JSON数据

3. 代码解析

3.1 请求头和Cookies

在向平台发送请求时，我们首先设置了请求头和Cookies，以模拟浏览器访问。

headers = {
    "Accept": "*/*",
    "Accept-Language": "zh-CN,zh;q=0.9",
    "Connection": "keep-alive",
    "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
    "Origin": "https://www.aqistudy.cn",
    "Referer": "https://www.aqistudy.cn/html/city_realtime.php?v=2.3",
    "Sec-Fetch-Dest": "empty",
    "Sec-Fetch-Mode": "cors",
    "Sec-Fetch-Site": "same-origin",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
    "X-Requested-With": "XMLHttpRequest",
    "sec-ch-ua": "\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"Windows\""
}

cookies = {
    "Hm_lvt_6088e7f72f5a363447d4bafe03026db8": "1723736791",
    "HMACCOUNT": "E6CE4DBDB92CCAA2",
    "Hm_lpvt_6088e7f72f5a363447d4bafe03026db8": "1723883137"
}
def read_js():
    with open("./spider.js", "r", encoding="utf-8") as f:
        return f.read()

def get_data(city_name):
    headers = {
        # 上述请求头部分
    }
    cookies = {
        # 上述cookies部分
    }
    url = "https://www.aqistudy.cn/apinew/aqistudyapi.php"
    js_file = read_js()
    obj = {"city": city_name}
    hXhY1B2Kd = execjs.compile(js_file).call("get_param", "GETDATA", obj)
    data = {
        "hXhY1B2Kd": hXhY1B2Kd
    }
    response = requests.post(url, headers=headers, cookies=cookies, data=data)
    print(response.text)
    result = decode_data(response.text)
    result_json = json.loads(result)
    rows_list = result_json['result']['data']['rows']
    for item in rows_list:
        print(item)

def decode_data(data):
    '''
    解密
    :param data:
    :return:
    '''
    js_file = read_js()
    ctx = execjs.compile(js_file)
    return ctx.call("decode_data", data)

if __name__ == '__main__':
    get_data("北京")