利用cookie进行模拟登录并且抓取失败
首先是朋友发现每次对撞md5都要上网站登录然后进行对撞,感觉好麻烦,想写一个脚本,输入md5值直接输出
然后就上车了
1 模拟登录
老规矩,先要提交表单,进行抓包(我用的fiddler)进行抓包,看见了post的表单,但心血来潮,发现每次模拟登录都利用提交表单的形式好无聊,再加上前些日子写web,就想利cookie试试。
可以看出,这个cookie中,
CNZZDATA3819543的ntime是时间,
user相当于session,其他都一样,所以可以写出模拟登录的脚本了
import requests from bs4 import BeautifulSoup import time URL = 'http://www.xxx.com/' def get_html(url): session = requests.session() headers = {'User_Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) ' 'AppleWebKit/537.36 (KHTML, like Gecko) ' 'Chrome/30.0.1581.2 Safari/537.36'} cookies = {'ASP.NET_SessionId': "eqsrnjotcaj5qdf5kmrqwgpy", 'CNZZDATA3819543': "cnzz_eid=471312766-1484873928-&ntime=%d" % int(time.time()), 'FirstVisit': "", '_test': "1", 'comefrom': "http://www.xxx.com/login.aspx", 'key': "", 'user': "kPXxHtwrSPpCMgZoXs2VrPuwuuCUrDz7dLq5R3/DBEP59eqYGYFa23AZdDPP1KDR9" "rblhGp0HWbYVkOsCg3QoRwWHIQESmZi4KqRlXxfnuZcFsrEta5SwAmrrvhpNvK" "ghSMRdyV7PTmKuagc7m8IZQ=="}
返回结果,进行解析html就可以得到用户名邮箱:
之后就可以利用session进行GET或者POST了
2 入坑,登录后的,进行md5的查询,然后抓包
接着看表单
分析表单:
__EVENTTARGET,__EVENTARGUMENT 这两个值没什么用,每次的值都是""。
__VIEWSTATE 这个值很有用,它是一种加密算法,结合了你查询的加密值和某些我未找到的值作为参数的加密算法(这是我没有实现爬虫的墙)
__VIEWSTATEGENERATOR 这个从字面上理解就是上面那个viewstate的生成器,我猜它的某种加密算法(不管了,懒得看了)
ctl00$ContentPlaceHolder1$TextBoxInput 这就是我们输入的需要解密的值
ctl00$ContentPlaceHolder1$InputHashType 这是我们选择的它是通过了什么加密,默认好像是md5
后面的值也什么大用。
其实说白了只要__VIEWSTATE 和ctl00$ContentPlaceHolder1$TextBoxInput的值相对应并且匹配,那么就没问题了。
3 最后奉献出我失败的爬虫
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
#!/usr/bin/python # -*- coding: utf-8 -*- import requests from bs4 import BeautifulSoup import time URL = 'http://www.xxx.com/' def get_html(url): session = requests.session() headers = {'User_Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) ' 'AppleWebKit/537.36 (KHTML, like Gecko) ' 'Chrome/30.0.1581.2 Safari/537.36'} cookies = {'ASP.NET_SessionId': "eqsrnjotcaj5qdf5kmrqwgpy", 'CNZZDATA3819543': "cnzz_eid=471312766-1484873928-&ntime=%d" % int(time.time()), 'FirstVisit': "", '_test': "1", 'comefrom': "http://www.xxxx.com/login.aspx", 'key': "", 'user': "kPXxHtwrSPpCMgZoXs2VrPuxxuCUrDz7dLq5R3/DBEP59eqYGYFa23AZdDPP1KDR9" "rblhGp0HWbYVkOsCg3QoRwWHIQESmZi4KqRlXxfnuZcFsrEta5SwAmrrvhpNvK" "ghSMRdyV7PTmKuagc7m8IZQ=="} playloads = {'__EVENTTARGET': "", '__EVENTARGUMENT': "", '__VIEWSTATE': "XrQ+lfRMi82hZRL/drrjo0zDnT6/XJxrr0iphlxrVrNfVusZC2UHmQL5" "i4TbbaD8N6zKVxODMamXqkA0k7T1qoNfW9dRGs/V6mEptB90XdBB4Qj1" "n1jGG/iw+p7BW4oHPanh8mWCH3G5ZWuZM4TADQoGwOuXna0OWtVK/x8k00" "+zZEwKXi0vI2T9OrysyhkZ8msq/yashFfMyDo+Qwqb3jNJWl8n844E9Kmb4" "gcBuBmifviw7jvRJjpVQNqDH+Cbee7gMEvFK4rtKxKcCkxIGNvC46F59rl" "62EfVX81NFVSD0dhGNnF7kP0WRWpcXZRoXrxd2HFodv5beAw8Gwe7IRHr59" "T8/GmiS3KVRMDXMG9OgAg13mZv9f/LogkuNmPeiIVz9fBifx2D2kUdQQfT5x" "T0wbqoGQnWqeQcEYndUCp5lA8kCID4V8p0TR3EfrzAHPlxPh7be8yNHL8iHu" "50wgxJ6BD2W3VoeF3lOShhkpnHYAeQf7TLaCCPtKleCboctIO6dbcgt1KD6S" "UvJZyWuRRxz/CBAGNEr6piRudKOgnGl+W9nBfJDS4wl3ao3Y3Rvuon0YMz68" "o+Ef4FOExM300T51rL5HF5e8zyw+V68ISvXAoHJmhzt64j+ht0jOUzLI1UTXo" "MOg894gucdsH8VOpVNPO5F+4/03JHqi8R4cSHnFu9U9gYpnGBhIhZuzzyiLHj" "a3gqyHzehKBlWq53eOhXJH/IfVjGZ9ltjZHi9smWCMonqvZRTm0vD6nKCsQWi" "JILUzb8YrI7xzYgjHihSEyYc3qi9ze6uSwUdeJbQdqKiGVWMWt+gRxi7JZDae" "SMfN3NvavFtXdyBVyI1KFuP9LBYDYEH1RD6HXqVsblH4C1dIAq7yQnu4L20OzI" "E841MIiwLdQVAQ9aAwD3wqvPqoBJfqbkMBKQ7xSiDF+FSRacJ/IHOAJkMoqKJe4LY" "Csh0tPK1tK1pW7xF/X+PtQCQQ+Ldin76t3bpeY2KAQeF5cXEP94DIYydiJBfn4zJv+D" "QBzb0zRabwy5GBB1YDY9Fxiw34G1rB18yOlTwl2bpFnUArplpB0TwfjGkA7Up2MCrOy" "s6oDDdRn+1AQOETo7Ych274ymw+ThCzUrJeVNPf5/X2FJCJpqeH0TRCSs+0fxbaljihS9" "p3t1WqTxTHWKsh4TsZBQsn90kSItZS/dGYhNH/XUVombBi92AhUrokHqQC4b0mGdIRFRzg" "6l2lF4VfZbDfIayTgnZbT+N9RwcduCZCRWcUupLLcKnCZHuqd7WStG33dTk9IT/5q2xf57G" "fRDxslLzN1VIDn8Wtcl494OJPSPqr5+FB8mTs24UjM+6IwgVNstkJFIH1urQWl31TVUg" "nhtrIQEs4MpyeeUUwlV2CCfxP+JTGbZsuMHdd/RDwp9xH28dGQD0cikU8RlCut/XThG" "W10bPC2akAXO5xmACNBhY9XKvyMzg8D43AFa3xAxV+e9lwPhNHIQCX7c6m/t5rQztzM" "+TiraaMMGXZVyjFic757VcJHlU5We8r7lWsKBRbrqnIEV6JMi8dzmb5rLYbBbLI4N9Q" "DIwy5r0HKDmepTjhZY3DIFLkdO9RakjAoiFUs2e9h+wPxBQGQ+UbyWXzfSWa8hXKSGL" "kw774/Et5XfCPVaDBkqPPzKlX3QoV5ptuRuDCwzLdXpuBePhme64x09L9XOmIYFdaGJ" "MXjw/tKRTv6AFgGLvZyso+Ch9XLI/j5abcaLyC/nSUdsxexRPkV/wRB5pSsaau43nMn" "iMpuAVVxwryPTGnnAO38vl26BAo73jlvNvmP0Av22/3P+A2CmCcJt6S5bH7Jcw6S6HJ" "QXWDtnFGg6sYCi6mzvwmYFcBEeVzOKHJ8f7TxP7n5CbNjXWnBguSFL1UzH83DTcij6s+1lctI" "fw4NIN7NU5P+qInfSRvBH3754GAuSApuLZHOp/9k8fkkxlA==", '__VIEWSTATEGENERATOR': "CA0B0334", 'ctl00$ContentPlaceHolder1$TextBoxInput': "21232f297a57a5a743894a0e4a801fc3", 'ctl00$ContentPlaceHolder1$InputHashType': "md5", 'ctl00$ContentPlaceHolder1$Button1': "查询", 'ctl00$ContentPlaceHolder1$HiddenField1': "", 'ctl00$ContentPlaceHolder1$HiddenField2': "gnSxKhU+42ESHE0pCcCyudmYfvxVL2+w4IhvdkwT37OI/" "QODVV7mdVAN9puROPjh"} text = session.post(url, headers=headers, data=playloads, cookies=cookies).text session.close() return text def parser_html(text): soup = BeautifulSoup(text, 'html.parser') string_gen = soup.find('div', class_='main').find('table', id='table3').\ find('span', id='ctl00_ContentPlaceHolder1_LabelAnswer').strings #strings属性返回一个生成器, 生成器返回的是一个iterable result = list(string_gen)[0] return result if __name__ == '__main__': text = get_html(URL) print parser_html(text)