爬虫实现有道翻译(超详细)
采集网站链接:在线翻译_有道
1、打开网站和开发者工具:
使用浏览器访问有道翻译网页,按F12打开开发者工具,切换到network选项卡,如图1所示;
![](http://i0.hdslb.com/bfs/article/0add71b8c729856a3fa151fee7a3886ea67d1073.jpg@942w_515h_progressive.webp)
2、输入内容记录数据包:
在有道翻译的输入框中输入要翻译的内容,让network记录出翻译时传递的数据包,如图2所示;
![](http://i0.hdslb.com/bfs/article/e02c1adbe6286991a353652dbba63c5b694c18ae.jpg@942w_476h_progressive.webp)
3、找传递数据的数据包:
翻译传递的数据包不多,逐个点击数据包,查看哪个包是传递的翻译内容,如图3所示;
![](http://i0.hdslb.com/bfs/article/2e5d9e995cb3d8f7576fc942efeb917ccb2acf91.jpg@942w_476h_progressive.webp)
4、解析数据包:
4.1、查看步骤3找到的数据包的Headers,可以记录下请求的相关数据,如图4.1所示;
请求链接:https://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule
请求方式:POST
请求头:
{Accept: application/json, text/javascript, */*; q=0.01
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9
Connection: keep-alive
Content-Length: 252
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Cookie: hb_MA-AF8F-B2A48FFDAF15_source=www.baidu.com; _ga=GA1.2.764619632.1646633958; OUTFOX_SEARCH_USER_ID_NCOO=1003225340.3669895; OUTFOX_SEARCH_USER_ID="1340665044@10.110.96.158"; JSESSIONID=aaaiVmNuaBUTAbiSgH8-x; fanyi-ad-id=305426; fanyi-ad-closed=1; ___rl__test__cookies=1649209424904
Host: fanyi.youdao.com
Origin: https://fanyi.youdao.com
Referer: https://fanyi.youdao.com/
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36
X-Requested-With: XMLHttpRequest}
![](http://i0.hdslb.com/bfs/article/2113a1b0ec00e7869885912b49cfc0468132d27f.jpg@942w_476h_progressive.webp)
4.2、查看步骤3找到的数据包的Payload,查看请求携带参数,如图4.2所示;
params:
{
smartresult: dict
smartresult: rule
}
data:
{
i: 你好
from: AUTO
to: AUTO
smartresult: dict
client: fanyideskweb
salt: 16492094249122
sign: 694bd830b856e25ee73bd7c46b43e213
lts: 1649209424912
bv: a0d7903aeead729d96af5ac89c04d48e
doctype: json
version: 2.1
keyfrom: fanyi.web
action: FY_BY_REALTlME
}
![](http://i0.hdslb.com/bfs/article/0e09d43d1ec6a1bd0ba4a0268152f57ecaa51a3f.jpg@942w_476h_progressive.webp)
5、根据数据包的解析情况,现在使用Python爬虫对数据包进行模拟,创建一个Python文件,导入三方模块requests,定义url、请求头和请求参数(请求参数Params不需要定义,url中“?”后,一般跟的是Params,如果url没有“?”则需要定义该参数),请求头大致分析下哪些是不必要参数,删掉不必要参数,大致代码如下:
import requests
url = 'https://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule'
headers = {
"Referer": "https://fanyi.youdao.com/",
"Cookie": 'OUTFOX_SEARCH_USER_ID=-1124603977@10.108.162.139; JSESSIONID=aaamH0NjhkDAeAV9d28-x; OUTFOX_SEARCH_USER_ID_NCOO=1827884489.6445506; fanyi-ad-id=305426; fanyi-ad-closed=1; ___rl__test__cookies=1649216072438',
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
}
data = {
"i": "你好",
"from": "AUTO",
"to": "AUTO",
"smartresult": "dict",
"client": "fanyideskweb",
"salt": "16492094249122",
"sign": "694bd830b856e25ee73bd7c46b43e213",
"lts": "1649209424912",
"bv": "a0d7903aeead729d96af5ac89c04d48e",
"doctype": "json",
"version": "2.1",
"keyfrom": "fanyi.web",
"action": "FY_BY_REALTlME",
}
res = requests.post(url,headers=headers,data=data)
print(res.text)
运行的结果是:
{"errorCode":50}
说明该请求头或请求参数中还存在错误,这个请求头中除了cookie以外,也没有其他什么复杂的参数。那么我们在看看请求参数,请求参数中,i是输入的需要翻译的字符串,lts一眼就可以看出是个13位的时间戳,salt参数和lts相比多了一位,sign和bv看起来都是md5加密值,其他的参数好像都是一些固定参数了。
6、再次输入一个翻译字符串,获取到数据包,如图6所示,对比两次翻译的数据包,对比结果可以分析出,出了i意外,确实是有lts、salt、sign和bv四个参数需要模拟;
![](http://i0.hdslb.com/bfs/article/9c9addb3aba59f134f765a49d722f044937746a5.jpg@942w_476h_progressive.webp)
7、再次点下All选项卡->Name下带有翻译参数的数据包,点击该数据包Initiator选下栏的位置,如图7所示;
![](http://i0.hdslb.com/bfs/article/eef8a0a8cb0de63647d04535a161ffe81ac7021e.jpg@942w_476h_progressive.webp)
8、点击后会来到source选项卡,点击代码块左下角的{},将js代码进行格式化,按下ctrl + f,打开搜索,在步骤6分析出的四个参数中,从命令来看,sign是指前面,我们在搜索框中输入sign,可以看到找到15个结果,我们一直向下查找,到第6个,8391行时,出现了md5字样,这个位置非常有可能就是sign的生成方式,同时在该代码上方两行有ts,bv,salt等,那么基本可以确定就是这里生成的参数,我们在有疑似对象的js代码对面数据上点击,打上断点,如图8所示;
![](http://i0.hdslb.com/bfs/article/9bec5ad26ece7a0a818998b96cabb062ce5ba7d0.jpg@942w_476h_progressive.webp)
9、简单的读下这段js代码,大致理解为,bv是t赋值的,t是n.md5(navigator.appVersion),n是e("./jquery-1.7");,navigator.appVersion是什么意思我也不知道,百度:是值请求浏览器的版本信息,也就是headers中的User-Agent的意思那么bv的值基本上就可以确定为User-Agent(UA)的md5加密值,因为爬虫是模拟浏览器的,所以复制的是浏览器的UA,那么bv这个值应该是不会改变的,bv这个值,我们还是使用原来参数的值即可。
10、ts是r赋值的,r是"" + (new Date).getTime(),大致可以理解为获取当前时间的时间戳;
salt是i赋值的,i是r + parseInt(10 * Math.random(), 10),大致就是r的基础上后面拼接一个10以内的数,和步骤6分析的参数也对的上;
sign是n.md5("fanyideskweb" + e + i + "Ygy_4c=r#e#4EX^NUGUc5"),就是拼接字符串的md5值,但是现在不知道e值是什么,根据图8的断定,我们在输入框中再次输入翻译字符串,如图10所示,可以看到,e就是输入需要翻译的字符串,到此参数解析已经全部完成。
![](http://i0.hdslb.com/bfs/article/fd71cf1c40339ea97f7dcabf5b10788564fec565.jpg@942w_476h_progressive.webp)
11、根据上面分析,我们在Python中模拟出对应的参数,代码如下所示:
import requests,time,random,hashlib
i = '你好'
lts = str(int(time.time()*1000)) #生成时间戳
print(lts)
salt = lts + str(random.randint(0,10)) #使用时间戳和随机0-9拼接salt
print(salt)
sign_str = 'fanyideskweb' + i + salt + 'Ygy_4c=r#e#4EX^NUGUc5'
m = hashlib.md5()
m.update(sign_str.encode())
sign = m.hexdigest() #生成MD5加密的sign
print(sign)
url = 'https://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule'
headers = {
"Referer": "https://fanyi.youdao.com/",
"Cookie": 'OUTFOX_SEARCH_USER_ID=-1124603977@10.108.162.139; JSESSIONID=aaamH0NjhkDAeAV9d28-x; OUTFOX_SEARCH_USER_ID_NCOO=1827884489.6445506; fanyi-ad-id=305426; fanyi-ad-closed=1; ___rl__test__cookies=1649216072438',
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
}
data = {
"i": i,
"from": "AUTO",
"to": "AUTO",
"smartresult": "dict",
"client": "fanyideskweb",
"salt": salt,
"sign": sign,
"lts": lts,
"bv": "a0d7903aeead729d96af5ac89c04d48e",
"doctype": "json",
"version": "2.1",
"keyfrom": "fanyi.web",
"action": "FY_BY_REALTlME",
}
print(data)
res = requests.post(url,headers=headers,data=data)
print(res.text)
通过动态模拟参数,可以获取到响应数据,如图11所示:
![](http://i0.hdslb.com/bfs/article/2a1573af4348970ecc722537f27c24dc6bbc2f87.png@942w_92h_progressive.webp)
12、最后将所有的代码规整一下,输入需要翻译的文本,打印出翻译的内容,最后贴上代码:
import requests,time,random,hashlib,json
i = input('请输入要需要翻译的文本:')
lts = str(int(time.time()*1000))
salt = lts + str(random.randint(0,9))
sign_str = 'fanyideskweb' + i + salt + 'Ygy_4c=r#e#4EX^NUGUc5'
m = hashlib.md5()
m.update(sign_str.encode())
sign = m.hexdigest()
url = 'https://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule'
headers = {
"Referer": "https://fanyi.youdao.com/",
"Cookie": 'OUTFOX_SEARCH_USER_ID=-1124603977@10.108.162.139; JSESSIONID=aaamH0NjhkDAeAV9d28-x; OUTFOX_SEARCH_USER_ID_NCOO=1827884489.6445506; fanyi-ad-id=305426; fanyi-ad-closed=1; ___rl__test__cookies=1649216072438',
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
}
data = {
"i": i,
"from": "AUTO",
"to": "AUTO",
"smartresult": "dict",
"client": "fanyideskweb",
"salt": salt,
"sign": sign,
"lts": lts,
"bv": "a0d7903aeead729d96af5ac89c04d48e",
"doctype": "json",
"version": "2.1",
"keyfrom": "fanyi.web",
"action": "FY_BY_REALTlME",
}
res = requests.post(url,headers=headers,data=data)
response = json.loads(res.text)
value = response['translateResult'][0][0]['tgt']
print(value)
实现效果如图12所示:
![](http://i0.hdslb.com/bfs/article/0c53f4a07221cf50869c8c10be85dbc10d71ba48.png@942w_107h_progressive.webp)