20170820_python实时获取某网站留言信息
主要用的是request和bs4,遇到最大的问题是目标站是gb2312编码,python3的编码虽然比2的处理要好得多但还是好麻烦,
最开始写的是用cookie模拟登陆,但是这个在实际使用中很麻烦,需要先登陆目标网站,然后把cookie复制下来拷贝到代码中...懒惰是
第一动力!
准备用火狐的httpfox获取下目标站post的数据和地址,发现火狐浏览器自动升级到了55.x,插件只能用在35.x版本,然后用chrome发现这
个网站提交post请求是打开了一个新的页面,然后新页面再点F12就晚了,看不到post了,然后百度一番发现可以设置新标签页开启F12!如图:
然后就知道了这个网站都post了什么数据,开始用requests模拟post,但是发现每次都登录失败,而且抓取的网页内容都是乱码,用了str('info', encoding='utf-8')才有所好转
发现根本就没有登录成功,然后提示输入账号密码登录。
灵光一闪!!!
估计是我post的数据是utf8而目标站接收post时是gb2312,根本看不懂啊!果断把用户名(用户名是中文!!!) username.encode("gb2312")之后顺利登录成功!然后又
开启了session保持cookie,持久化登录。然后每分钟判断下最后一个id是否等于保存的id,判断是否进行抓取。
效果如下:
#-*-coding:utf-8-*- #编码声明 import requests,re,time,json,os from bs4 import BeautifulSoup from time import strftime,gmtime LOGIN_URL = 'http://www.3456.tv/Default.aspx' #请求的URL地址 username = '用户名' password = 'password' DATA = {"web_top_two2$txtName":username.encode("gb2312"), "web_top_two2$txtPass":password, '__VIEWSTATE':'/wEPDwULLTEyNzc4MjM2OTBkGAEFHl9fQ29udHJvbHNSZXF1aXJlUG9zdEJhY2tLZXlfXxYBBRh3ZWJfdG9wX3R3bzIkaW1nQnRuTG9naW6/pqbjQqV358GfYjdoiOK+Ek4VWA==','__EVENTVALIDATION':'/wEWBAL3y5PLCgLHgt+5BgL3r9v/CgLX77PND5R1XxTeGn4lXvBDrb6OdRyc4Xlk','web_top_two2$imgBtnLogin.x':'22','web_top_two2$imgBtnLogin.y':'8'} #登录系统的账号密码,也是我们请求数据 HEADERS = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36'} #模拟登陆的浏览器 S = requests.Session() login = S.post(LOGIN_URL,data=DATA,headers=HEADERS) #模拟登陆操作 def getData(num): url = 'http://www.3456.tv/user/list_proxyall.html' res = S.get(url) content = res.content return content def getLast(): url = 'http://www.3456.tv/user/list_proxyall.html' res = S.get(url) content = res.content soup = BeautifulSoup(content,'html.parser') tb = soup.find_all('tr',style='text-align:center;') for tag in tb: see = tag.find('a', attrs={'class':'see'}) seestr = see['onclick'] seenum = re.sub("\D", "", seestr) break return seenum def isNew(): newlastid = getLast() with open('lastid.txt') as txt: last = txt.read() if int(newlastid) != int(last): print('当前时间:' + strftime("%H-%M") + ',发现新留言,获取中!') getNewuser() else: print('当前时间:' + strftime("%H-%M") + ',暂时没有新留言') def getNewuser(): url = 'http://www.3456.tv/user/list_proxyall.html' res = S.get(url) content = res.content soup = BeautifulSoup(content,'html.parser') tb = soup.find_all('tr',style='text-align:center;') with open('lastid.txt') as txt: last = txt.read() userinfo = '' for tag in tb: see = tag.find('a', attrs={'class':'see'}) seestr = see['onclick'] seenum = re.sub("\D", "", seestr) if int(seenum) == int(last): break userinfo += (str(seeInfo(int(seenum)), encoding = "utf-8") + '\n') userfilename = strftime("%H-%M") + '.txt' with open( userfilename, 'w') as f: f.write(str(userinfo)) os.system(userfilename) with open('lastid.txt', 'w') as f2: f2.write(str(getLast())) print('本次抓取完成,当前时间:' + strftime("%H-%M") + ',60秒后继续执行') def seeInfo(id): url = 'http://www.3456.tv/user/protel.html' info = {'id':id} res = S.get(url,data=info) content = res.content return content setsleep = 60 #修改这个设置每次抓取间隔,60为60秒 print('this time is today first time start?') firststr = input('input yes or no and press enter: ') if firststr == 'yes': print('正在抓取中...') lastid = getLast() with open('lastid.txt', 'w') as f: f.write(str(lastid)) print('当前时间:' + strftime("%H:%M") + ',当前第一条数据id为' + lastid) print(str(setsleep) + '秒后继续执行') else: print(str(setsleep) + '秒后继续执行') while 1: isNew() time.sleep(int(setsleep))