python爬虫-requests
requests简介
requests是对于urllib有着很多优势,它能够更好的处理关于cookies,登录验证,代理设置等操作而不需要想urllib南无麻烦,以下为requests经常用到的一些方法
基本用法
get()方法请求网页,实现与urllib库urlopen()方法相同的操作,得到一个Response对象,以及这个对象的类型与属性
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests r = requests.get('https://www.baidu.com/') print(type(r)) print(r.status_code) print(type(r.text)) print(r.cookies) 结果: #得到一个Response对象 <class 'requests.models.Response'> #状态码 200 #响应体类型 <class 'str'> #cookies <RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>
Get请求请求链接为http://httpbin.org/get,测试客户端是否发起Get请求,而且会返回相应的请求信息,其格式为json格式。
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests r = requests.get('http://www.httpbin.org/get') print(r.text) 结果: { "args": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Connection": "close", "Host": "www.httpbin.org", "User-Agent": "python-requests/2.19.1" }, "origin": "36.40.49.173", "url": "http://www.httpbin.org/get" }
json格式转换为字典格式
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests r = requests.get('http://www.httpbin.org/get') print(type(r.text)) print(type(r.json())) print(r.json()) 结果: <class 'str'> <class 'dict'> {'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Host': 'www.httpbin.org', 'User-Agent': 'python-requests/2.19.1'}, 'origin': '36.40.49.173', 'url': 'http://www.httpbin.org/get'}
Get请求抓取知乎网页(普通网页信息一般不为json格式),(需要传入header信息,即User-Agent字段信息),否则知乎禁止抓取
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests import re headers = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0' } r = requests.get('https://www.zhihu.com/explore',headers = headers) pattern = re.compile('explore-feed.*?question_link.*?>(.*?)</a>',re.S) titles = re.findall(pattern,r.text) print(titles) 结果: ['\n地理这门学科有多有趣?\n', '\n如何看待辩论赛收报名费的现象?\n', '\n如何评价朱一龙?\n', '\n历史上哪些人的名或字比较奇怪?\n', '\n有哪些惊艳到你的句子?\n', '\n英雄联盟中有哪些冷知识?\n', '\n如何评价《海贼王》第925话?\n', '\n怎样看待华晨宇说自己做音乐的天赋占百分之九十九,努力占百分之一?\n', '\n你觉得《三体》中最残忍的一句话是什么?\n', '\n人类有哪些细思恐极的事?\n']
抓取二进制数据(图片,音频,视频等)并保存至文件:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests r = requests.get('https://github.com/favicon.ico') #返回结果乱码 print(r.text) #返回二进制数据 print(r.content) 部分结果: �������O L������ b'\x00\x00\x01\x00\x02\x00\x10\x10\x00\x00\x01\x00 \x00(\x05\x00\x00&\x00\x00\x00 \x00\x00\x01\x00 图片保存至文件: import requests r = requests.get('https://www.github.com/favicon.ico') with open('favicon.ico','wb') as f: f.write(r.content)
requests post 请求:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests data = { 'name' : 'getmey', 'age' : 22 } r = requests.post('http://httpbin.org/post',data=data) print(r.text) 结果: { "args": {}, "data": "", "files": {}, "form": { "age": "22", "name": "getmey" }, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Connection": "close", "Content-Length": "18", "Content-Type": "application/x-www-form-urlencoded", "Host": "httpbin.org", "User-Agent": "python-requests/2.19.1" }, "json": null, "origin": "36.40.49.173", "url": "http://httpbin.org/post" }
响应信息:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests r = requests.get('http://www.jianshu.com') print(type(r.status_code),r.status_code) print(type(r.headers),r.headers) print(type(r.cookies),r.cookies) print(type(r.url),r.url) print(type(r.history),r.history) 结果: <class 'int'> 403 <class 'requests.structures.CaseInsensitiveDict'> {'Date': 'Mon, 24 Dec 2018 02:36:55 GMT', 'Content-Type': 'text/html', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Server': 'Tengine', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Content-Encoding': 'gzip', 'X-Via': '1.1 PSbjwjBGP2yt134:5 (Cdn Cache Server V2.0), 1.1 PSzjwzdx11at80:10 (Cdn Cache Server V2.0), 1.1 PSsxwndx4au44:4 (Cdn Cache Server V2.0)'} <class 'requests.cookies.RequestsCookieJar'> <RequestsCookieJar[]> <class 'str'> https://www.jianshu.com/ <class 'list'> [<Response [301]>]
Requests高级用法
文件上传
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests files = {'file' : open('favicon.ico','rb')} r = requests.post('http://httpbin.org/post',files=files) print(r.text) 结果: { "args": {}, "data": "", "files": { "file": "data:application/octet-stream;base64,AAABAAIAEBAAAAEAIAA..... } "form": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Connection": "close", "Content-Length": "6665", "Content-Type": "multipart/form-data; boundary=aa294e10346d7b2538cf2d744dd46855", "Host": "httpbin.org", "User-Agent": "python-requests/2.19.1" }, "json": null, "origin": "36.40.49.173", "url": "http://httpbin.org/post" } Cookies:
Cookies 获取
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests r = requests.get('https://www.baidu.com') print(r.cookies) 结果: <RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>
会话维持-Session用于模拟在一个浏览器中打开同一个站点的不同页面
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests s = requests.Session() s.get('http://httpbin.org/cookies/set/number/123456789') r = s.get('http://httpbin.org/cookies') print(r.text) 结果: { "cookies": { "number": "123456789" } }
代理设置
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests proxies = { 'http' : 'http://10.10.1.10:3128', 'https' : 'http://10.10.1.10:1080' } 或 proxies ={ 'http': 'socket5://user:password@host:port', 'https': 'socket5://user:password@host:por' } 或 proxies = { 'http' : 'http://user:password@10.10.1.10:3128' } requests.get('https://www.taobao.com',proxies=proxies)
超时设置
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests #链接和读取之和为1秒 r = requests('https://www.taobao.com',timeout=1) print(r.status_code) import requests #分别设置链接和读取时间 r = requests.get('https://www.taobao.com',timeout=(5,10)) print(r.status_code) import requests #timeout设置为None,永久等待 r = requests.get('https://www.taobao.com',timeout=None) print(r.status_code)
身份认证:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
import requests from requests.auth import HTTPBasicAuth #具体以实际情况为准 r = requests.get("http:localhost:5000",auth=HTTPBasicAuth('username','password')) print(r.status_code)
Prepared Request-将请求表示为数据结构且数据结果就称为Prepared Request
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
from requests import Request,Session url = 'http://httpbin.org/post' data ={ 'name' : 'getmey', } headers = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0' } s = Session() req = Request('POST',url,data=data,headers=headers) preped = s.prepare_request(req) r = s.send(preped) print(r.text) 结果: { "args": {}, "data": "", "files": {}, "form": { "name": "getmey" }, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Connection": "close", "Content-Length": "11", "Content-Type": "application/x-www-form-urlencoded", "Host": "httpbin.org", "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0" }, "json": null, "origin": "36.40.49.173", "url": "http://httpbin.org/post" }