Python 爬虫知识点 - 淘宝商品检索结果抓包分析(续二)
一、URL分析
通过对“Python机器学习”结果抓包分析,有两个无规律的参数:_ksTS和callback。通过构建如下URL可以获得目标关键词的检索结果,如下所示:
https://s.taobao.com/search?data-key=s&data-value=44&ajax=true&_ksTS=1482325509866_2527&callback=jsonp2528&q=Python机器学习&imgfile=&js=1&stats_click=search_radio_all:1&initiative_id=staobaoz_20161221&ie=utf8&bcoffset=0&ntoffset=6&p4ppushleft=.44&fs=1&s=0
https://s.taobao.com/search?data-key=s&data-value=88&ajax=true&_ksTS=1482325509866_2527&callback=jsonp2528&q=Python机器学习&imgfile=&js=1&stats_click=search_radio_all:1&initiative_id=staobaoz_20161221&ie=utf8&bcoffset=0&ntoffset=6&p4ppushleft=.44&fs=1&s=44
https://s.taobao.com/search?data-key=s&data-value=132&ajax=true&_ksTS=1482325509866_2527&callback=jsonp2528&q=Python机器学习&imgfile=&js=1&stats_click=search_radio_all:1&initiative_id=staobaoz_20161221&ie=utf8&bcoffset=0&ntoffset=6&p4ppushleft=.44&fs=1&s=88
https://s.taobao.com/search?data-key=s&data-value=176&ajax=true&_ksTS=1482325509866_2527&callback=jsonp2528&q=Python机器学习&imgfile=&js=1&stats_click=search_radio_all:1&initiative_id=staobaoz_20161221&ie=utf8&bcoffset=0&ntoffset=6&p4ppushleft=.44&fs=1&s=132
https://s.taobao.com/search?data-key=s&data-value=220&ajax=true&_ksTS=1482325509866_2527&callback=jsonp2528&q=Python机器学习&imgfile=&js=1&stats_click=search_radio_all:1&initiative_id=staobaoz_20161221&ie=utf8&bcoffset=0&ntoffset=6&p4ppushleft=.44&fs=1&s=176
https://s.taobao.com/search?data-key=s&data-value=264&ajax=true&_ksTS=1482325509866_2527&callback=jsonp2528&q=Python机器学习&imgfile=&js=1&stats_click=search_radio_all:1&initiative_id=staobaoz_20161221&ie=utf8&bcoffset=0&ntoffset=6&p4ppushleft=.44&fs=1&s=220
https://s.taobao.com/search?data-key=s&data-value=308&ajax=true&_ksTS=1482325509866_2527&callback=jsonp2528&q=Python机器学习&imgfile=&js=1&stats_click=search_radio_all:1&initiative_id=staobaoz_20161221&ie=utf8&bcoffset=0&ntoffset=6&p4ppushleft=.44&fs=1&s=264
https://s.taobao.com/search?data-key=s&data-value=352&ajax=true&_ksTS=1482325509866_2527&callback=jsonp2528&q=Python机器学习&imgfile=&js=1&stats_click=search_radio_all:1&initiative_id=staobaoz_20161221&ie=utf8&bcoffset=0&ntoffset=6&p4ppushleft=.44&fs=1&s=308
二、关键字分析
1、q查询关键词
2、data-value显示记录数
3、s上一页记录数
4、s与data-value的差值即当页显示数量
三、Python抓取数据
#__author__ = 'Joker'
# -*- coding:utf-8 -*-
import re
import urllib.request
keyWord1 = "Python机器学习"
keyWord2 = urllib.request.quote(keyWord1)
headers = ("User-Agent","MMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.1708.400 QQBrowser/9.5.9635.400")
opener = urllib.request.build_opener()
opener.addheaders = [headers]
urllib.request.install_opener(opener)
for j in range(1,25):
try:
curPage = 44
prePage = 0
url = "https://s.taobao.com/search?data-key=s&data-value=" + str(
curPage) + "&ajax=true&_ksTS=1482325509866_2527&callback=jsonp2528&q=" + keyWord2 + "&imgfile=&js=1&stats_click=search_radio_all:1&initiative_id=staobaoz_20161221&ie=utf8&bcoffset=0&ntoffset=6&p4ppushleft=.44&fs=1&s=" + str(
prePage)
data = urllib.request.urlopen(url).read().decode("utf-8", "ignore")
patTitle = '"title":"(.*?)","raw_title"'
titles = re.compile(patTitle).findall(data)
patRawTitle = '"raw_title":"(.*?)"'
rawTitles = re.compile(patRawTitle).findall(data)
patImage = '"pic_url":"//(.*?)","'
rawImages = re.compile(patImage).findall(data)
patPrice = '"view_price":"(.*?)","'
rawPrices = re.compile(patPrice).findall(data)
patNick = '"nick"(.*?)","'
rawNicks = re.compile(patNick).findall(data)
for i in range(0,len(titles)):
print("-------------------")
print("第" + str(j+1) + "页,第" + str(i+1) + "本" )
#print(titles[i])
print(rawTitles[i])
print(rawImages[i])
print(rawPrices[i])
print(rawNicks[i])
print("-------------------")
prePage = 44 * j
curPage = 44 + prePage
except urllib.error.URLError as e:
if hasattr(e,"code"):
print(e.code)
if hasattr(e,"reason"):
print(e.reason)
except Exception as e:
print(e)
博客版权: 本文以学习、研究和分享为主,欢迎转载和各类爬虫,但必须在文章页面明显位置给出原文链接。 如果文中有不妥或者错误的地方还望高手的您指出,以免误人子弟。如果您有更好的建议,不如留言一起讨论,共同进步! 再次感谢您耐心的读完本篇文章。
【推荐】国内首个AI IDE,深度理解中文开发场景,立即下载体验Trae
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· Linux系列:如何用 C#调用 C方法造成内存泄露
· AI与.NET技术实操系列(二):开始使用ML.NET
· 记一次.NET内存居高不下排查解决与启示
· 探究高空视频全景AR技术的实现原理
· 理解Rust引用及其生命周期标识(上)
· 物流快递公司核心技术能力-地址解析分单基础技术分享
· .NET 10首个预览版发布:重大改进与新特性概览!
· 单线程的Redis速度为什么快?
· 展开说说关于C#中ORM框架的用法!
· Pantheons:用 TypeScript 打造主流大模型对话的一站式集成库