异步HTTP请求- GRequests

一. 介绍：

　　1. GRequests是一个基于Python的HTTP库，它是Requests库的扩展。与Requests库不同的是，GRequests允许开发人员通过协程的方式并发发送HTTP请求，从而提高程序的性能。GRequests还提供了一个方便的线程池和协程池，可以更好地控制并发请求的数量。

　　2. 官方文档地址：https://pypi.org/project/grequests/

 

二. 使用：

　　1. 初步使用

　　　　

import grequests


urls = [
    'https://www.baidu.com/',
    'https://fanyi.baidu.com/translate',
    'https://note.youdao.com/web',
    'https://www.yuque.com/',
    'https://www.cnblogs.com/aggsite/allsitecategories',

]

# 创建一组未发送的请求
res = (grequests.get(u) for u in urls)
# 同时发送它们
response = grequests.map(res)
print(response)

 

　　2. 使用回调函数

def print_response(response, *args, **kwargs):
    # print(response.content)
    print(response.url)
    print(response.text)


urls = ['http://httpbin.org/delay/1']*3
requests = [grequests.get(url, callback=print_response) for url in urls]
responses = grequests.map(requests)

 

 　　3. 使用会话-会话可以在多个请求之间共享状态，例如cookie或身份验证信息

session = grequests.Session()
urls = ['http://httpbin.org/get', 'http://httpbin.org/post']
requests = [grequests.get(url, session=session) for url in urls]
responses = grequests.map(requests)

 

　　4. 使用 自定义请求的URL、请求头和请求参数

urls = ['https://www.cnblogs.com/aggsite/allsitecategories', 'https://www.cnblogs.com/hejiale010426/ajax/blogStats']
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}
# params = {'usernme': 'xx', 'password': ''}

session = grequests.Session()
# 创建请求列表
requests = (grequests.get(u, headers=headers, params='') for u in urls)

# 发送请求并获取响应
responses = grequests.map(requests)

# 处理响应
for response in responses:
    if response is not None:
        # print(response.url)
        # print(response.text)
        print(response)
    else:
        print('Request failed')