9 chrome能打开去哪儿的机票页面而python selenium启动的chrome不行 2
------------恢复内容开始------------
------------恢复内容开始------------
https://flight.qunar.com/site/oneway_list.htm?searchDepartureAirport=%E4%B8%8A%E6%B5%B7&searchArrivalAirport=%E5%8C%97%E4%BA%AC&searchDepartureTime=2021-10-18&searchArrivalTime=2021-10-21&nextNDays=0&startSearch=true&fromCode=SHA&toCode=BJS&from=qunarindex&lowestPrice=null
上面这一串地址复制到chrome浏览器和别的浏览器比如qq浏览器中,都能访问。唯独从selenium中启动的chrome中,会变成如下界面。无法显示2021-10-21日的机票信息。
初步怀疑是selenium不支持我当前的chrome版本。(更新:2021年10月17日10:47:22 后来降低为selenium支持的92.x.x.107版本,包括chrome driver也降了。还是不行)
现象就是:直接启动chrome能搜索去哪儿,并且显示上海到北京的次日机票信息。而从python程序中selenium启动的chrome中,死活打不开。然后把地址复制出来到任何浏览器,包括没有selenium控制的chrome,都能打开网页。
具体原因待查。
为什么要用selenium
首先,机票页面是XHR中用js交互产生的动态渲染数据,因此直接抓取页面的源代码中,没有任何机票信息。只有用selenium,所见即所可得。
有人会说,那么就模拟XHR请求。
我通过在postman中模拟XHR请求,
返回的是请求成功不错。但是没有任何数据。
而浏览器中访问中返回的数据是有机票信息的。
更新:
后续通过补全headers信息,发现就可以获取返回数据了。下面是补全的headers信息。
下面是传递的get参数信息,就是直接在url后面的。
获取的返回数据是:
并且这里面包括页面中后面页面的数据。也就是点击第2页和下一页才会显示的数据。
并且,在headers中隔了一天以后,要重新打开浏览器,修改pre参数。这个pre应该是一种验证机制。如果pre不修改,之前的请求,隔了一天,无法请求到机票数据。
那么selenium,所见即所可得,能否获取js动态渲染中下一页才显示的数据呢?
不仅chrome,Firefox也遇到同样问题
import time from selenium.webdriver import Firefox from selenium.webdriver.firefox.options import Options from selenium.webdriver.firefox.firefox_binary import FirefoxBinary from selenium.webdriver.common.desired_capabilities import DesiredCapabilities import requests,json options = Options() # options.binary_location = "C:\\Users\\xiaojie\\AppData\\Local\\Google\\Chrome\\Application\\chrome.exe" options.binary_location = "C:\\Program Files\\Mozilla Firefox\\firefox.exe" binary= FirefoxBinary("C:\\Program Files\\Mozilla Firefox\\firefox.exe") caps = DesiredCapabilities.FIREFOX.copy() caps['marionette'] = True # options.add_experimental_option('excludeSwitches', ['enable-automation']) # options.add_argument('--incognito') # options.add_argument('disable-infobars') # options.add_argument('log-level=3') driver =Firefox(firefox_binary=binary,capabilities=caps, executable_path="geckodriver.exe") url="https://www.qunar.com/" # url = "https://diannao.jd.com/" driver.get(url) time.sleep(1) url="https://flight.qunar.com/site/oneway_list.htm?searchDepartureAirport=%E4%B8%8A%E6%B5%B7&searchArrivalAirport=%E5%8C%97%E4%BA%AC&searchDepartureTime=2021-10-18&searchArrivalTime=2021-10-22&nextNDays=0&startSearch=true&fromCode=SHA&toCode=BJS&from=qunarindex&lowestPrice=null" #搜索机票 #两种方式 driver.get(url) with open('items.jl','w',encoding='UTF-8') as file: file.write(driver.page_source) url = "https://diannao.jd.com/" #设置header属性 header={ "User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3877.400 QQBrowser/10.8.4506.400" ,"Referer":"http://m.611.com/Match/Index" ,"Connection":"keep-alive" ,"Cookie":"__jdu=1961246440; shshshfpa=9d816ce7-9076-04a7-ea7d-8018bc6eadf2-1623898299; shshshfpb=sdus9TfEminz9dWA7KKGYgw%3D%3D; pinId=pjKhXUE59i7LjgxtUlkd_A; pin=zyj183247166; unick=jdzyj183; _tp=LdLet8T0koyg96E1dqQafA%3D%3D; _pst=zyj183247166; areaId=2; TrackID=1CmX15GEs1MOTf99XeZl5eebqfftNFiLEiMsS8vvKBTMwCAfRlKqXu7YCjQn__C2-mqlg-FJxPlEwiA79snSf04SU1xTtsZOoj5aQk_Cb5mu1XN52nGptNsMI-kJjYqCV; user-key=03e76034-3bcd-4674-a834-cf34a4b960a6; ipLocation=%u4e0a%u6d77; cn=76; ipLoc-djd=2-2824-61056-0.3405425761; unpl=V2_ZzNtbUVQQhV1DhJXLxhfV2IFFV0RAkoSJltEAHsRWwc1AUdbclRCFnUURlVnGVsUZgsZXkJcQxFFCEdkeBBVAWMDE1VGZxBFLV0CFSNGF1wjU00zQwBBQHcJFF0uSgwDYgcaDhFTQEJ2XBVQL0oMDDdRFAhyZ0AVRQhHZHsRWwVkBhVYR1ZzJXI4dmR%2fHV4BbwciXHJWc1chVEBTeRBdByoDGlpCVEYScA1HZHopXw%3d%3d; __jdv=76161171|baidu-pinzhuan|t_288551095_baidupinzhuan|cpc|0f3d30c8dba7459bb52f2eb5eba8ac7d_0_660117e2e02c4761bd86bb3e1963c3d7|1634347276899; PCSYCityID=CN_310000_310100_310101; shshshfp=26c4ea20d915727cb0bd98f196346b4d; __jdc=122270672; wlfstk_smdl=7bqz560g1rixn5kg0i84v5y6xvvrqkhx; __jda=122270672.1961246440.1623769658.1634347277.1634382193.23; __jdb=122270672.1.1961246440|23.1634382193; o2-webp=true; 3AB9D23F7A4B3C9B=TJ6FIT6PTK3N32QTINQHHUBRA4J4MDPCZWEHCIIXVS6J5H3LSD75C3RMTC2RIBLHQDLJOCOMXWMJ2LSD6IEMJMV66M" } #response = requests.get("http://m.611.com/Match/Index",headers=header) response = requests.get(url,headers=header) if response.status_code == 200: print(response.text) with open('items2.jl','w') as file: file.write(response.text) # data = json.loads(response.text) # token = data["Data"] time.sleep(4) #商品信息 item__info=driver.find_elements_by_class_name('goods-item') for item in item__info: name=item.find_element_by_class_name('goods-item__info').text price=item.find_element_by_class_name('goods-item__price').text print(name) print(price) print("-----------") driver.delete_all_cookies() # driver.quit()
使用selenium启动firefox访问机票信息。同样页面卡机
可能原因就是去哪儿机票网,设置了对自动化测试工具webdriver的反爬。导致无法爬取。同样的链接,直接启动firefox或者chrome浏览器,都能爬取。唯独通过selenium启动后,不能访问网页。
最后实验出来是因为设置了针对selenium的反爬
对于网上说的mitmproxy设置代理的方法,较为复杂。
最后添加了两个参数,隐藏selenium的特征后,就能正常使用selenium进行爬取了。
完整代码如下:
import time # from selenium.webdriver import Firefox # from selenium.webdriver.firefox.options import Options from selenium.webdriver import Chrome from selenium.webdriver.chrome.options import Options # from selenium.webdriver.firefox.firefox_binary import FirefoxBinary # from selenium.webdriver.common.desired_capabilities import DesiredCapabilities import requests,json options = Options() options.binary_location = "C:\\Users\\xiaojie\\AppData\\Local\\Google\\Chrome\\Application\\chrome.exe" options.add_experimental_option('excludeSwitches', ['enable-automation']) options.add_argument('--incognito') options.add_argument('disable-infobars') options.add_argument('log-level=3') options.add_argument("--disable-blink-features") options.add_argument("--disable-blink-features=AutomationControlled") # options.binary_location = "C:\\Program Files\\Mozilla Firefox\\firefox.exe" # binary= FirefoxBinary("C:\\Program Files\\Mozilla Firefox\\firefox.exe") # caps = DesiredCapabilities.FIREFOX.copy() # caps['marionette'] = True # driver =Firefox(firefox_binary=binary,capabilities=caps, executable_path="geckodriver.exe") driver =Chrome(options=options,executable_path="D:\\webdriver\\chromedriver_win32\\chromedriver.exe") # script = 'Object.defineProperty(navigator,"webdriver",{get:()=>false,});' # driver.execute_script(script) url="https://www.qunar.com/" # url = "https://diannao.jd.com/" driver.get(url) time.sleep(1) url="https://flight.qunar.com/site/oneway_list.htm?searchDepartureAirport=%E4%B8%8A%E6%B5%B7&searchArrivalAirport=%E5%8C%97%E4%BA%AC&searchDepartureTime=2021-10-18&searchArrivalTime=2021-10-22&nextNDays=0&startSearch=true&fromCode=SHA&toCode=BJS&from=qunarindex&lowestPrice=null" #搜索机票 #两种方式 # script = 'Object.defineProperty(navigator,"webdriver",{get:()=>false,});' # driver.execute_script(script) driver.get(url) # script = 'Object.defineProperty(navigator,"webdriver",{get:()=>false,});' # driver.execute_script(script) with open('items.jl','w',encoding='UTF-8') as file: file.write(driver.page_source) url = "https://diannao.jd.com/" #设置header属性 header={ "User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3877.400 QQBrowser/10.8.4506.400" ,"Referer":"http://m.611.com/Match/Index" ,"Connection":"keep-alive" ,"Cookie":"__jdu=1961246440; shshshfpa=9d816ce7-9076-04a7-ea7d-8018bc6eadf2-1623898299; shshshfpb=sdus9TfEminz9dWA7KKGYgw%3D%3D; pinId=pjKhXUE59i7LjgxtUlkd_A; pin=zyj183247166; unick=jdzyj183; _tp=LdLet8T0koyg96E1dqQafA%3D%3D; _pst=zyj183247166; areaId=2; TrackID=1CmX15GEs1MOTf99XeZl5eebqfftNFiLEiMsS8vvKBTMwCAfRlKqXu7YCjQn__C2-mqlg-FJxPlEwiA79snSf04SU1xTtsZOoj5aQk_Cb5mu1XN52nGptNsMI-kJjYqCV; user-key=03e76034-3bcd-4674-a834-cf34a4b960a6; ipLocation=%u4e0a%u6d77; cn=76; ipLoc-djd=2-2824-61056-0.3405425761; unpl=V2_ZzNtbUVQQhV1DhJXLxhfV2IFFV0RAkoSJltEAHsRWwc1AUdbclRCFnUURlVnGVsUZgsZXkJcQxFFCEdkeBBVAWMDE1VGZxBFLV0CFSNGF1wjU00zQwBBQHcJFF0uSgwDYgcaDhFTQEJ2XBVQL0oMDDdRFAhyZ0AVRQhHZHsRWwVkBhVYR1ZzJXI4dmR%2fHV4BbwciXHJWc1chVEBTeRBdByoDGlpCVEYScA1HZHopXw%3d%3d; __jdv=76161171|baidu-pinzhuan|t_288551095_baidupinzhuan|cpc|0f3d30c8dba7459bb52f2eb5eba8ac7d_0_660117e2e02c4761bd86bb3e1963c3d7|1634347276899; PCSYCityID=CN_310000_310100_310101; shshshfp=26c4ea20d915727cb0bd98f196346b4d; __jdc=122270672; wlfstk_smdl=7bqz560g1rixn5kg0i84v5y6xvvrqkhx; __jda=122270672.1961246440.1623769658.1634347277.1634382193.23; __jdb=122270672.1.1961246440|23.1634382193; o2-webp=true; 3AB9D23F7A4B3C9B=TJ6FIT6PTK3N32QTINQHHUBRA4J4MDPCZWEHCIIXVS6J5H3LSD75C3RMTC2RIBLHQDLJOCOMXWMJ2LSD6IEMJMV66M" } #response = requests.get("http://m.611.com/Match/Index",headers=header) response = requests.get(url,headers=header) if response.status_code == 200: print(response.text) with open('items2.jl','w') as file: file.write(response.text) # data = json.loads(response.text) # token = data["Data"] time.sleep(4) #商品信息 item__info=driver.find_elements_by_class_name('goods-item') for item in item__info: name=item.find_element_by_class_name('goods-item__info').text price=item.find_element_by_class_name('goods-item__price').text print(name) print(price) print("-----------") driver.delete_all_cookies() # driver.quit()
里面添加了两句话:
options.add_argument("--disable-blink-features") options.add_argument("--disable-blink-features=AutomationControlled")
同时,发现selenium只能抓取当前页显示的数据。无法像前述模拟XHR请求,返回所有机票数据。
至于设置的文本混淆。则需要用其他方法解决。
但是模拟XHR请求,然后尚未渲染到页面的,直接从服务器返回的数据,是应对文本混淆的最佳方法。
------------恢复内容结束------------
------------恢复内容结束------------