使用Python3 协程语法 async await 来实现协程异步http请求
都知道Python的多任务有些尴尬，多进程可以用多核，但是消耗大，线程吧，无能用多核，是全局解释器锁来回切，所以通常都比较青睐协程了，但是协程是基于生成器的，不使用第三方库的开发成本学习成本就上去了，目前用的多的就是Gevent，基于Greenlet，使用类似于线程，不过在Python3.5以上版本Python提供了协程语法，可以更方便的使用，但是新的语法和一般的Python语法又有些不同，使用新的关键字，async，await
下面以一个例子来实现协程异步操作http请求
1 import asyncio 2 import traceback 3 import aiohttp 4 5 6 Normal = "http://github.com/" 7 8 async def get_url(url): 9 client = aiohttp.ClientSession() 10 try: 11 resp = await client.get(url) 12 await client.close() 13 return resp 14 except: 15 await client.close() 16 return traceback.format_exc() 17 18 async def post_url(url, data:dict=None): 19 async with aiohttp.ClientSession() as client: 20 try: 21 async with client.post(url, data=data) as resp: 22 return resp 23 except: 24 return traceback.format_exc() 25 26 async def ask_url(url=Normal): 27 print("%s===Will Await..."%url) 28 response = await get_url(url) 29 print("%s===Stop . . ."%url) 30 print(response) 31 32 33 if __name__ == '__main__': 34 urls = [Normal, "http://baidu.com", "http://zhihu.com"] 35 loop = asyncio.get_event_loop() 36 tasks = [asyncio.ensure_future(ask_url(x)) for x in urls] 37 loop.run_until_complete(asyncio.wait(tasks)) 38 loop.close()
执行结果如下：
http://github.com/===Will Await... http://baidu.com===Will Await... http://zhihu.com===Will Await... http://baidu.com===Stop . . . <ClientResponse(http://baidu.com) [200 OK]> <CIMultiDictProxy('Date': 'Sat, 19 Jun 2021 14:39:29 GMT', 'Server': 'Apache', 'Last-Modified': 'Tue, 12 Jan 2010 13:48:00 GMT', 'Etag': '"51-47cf7e6ee8400"', 'Accept-Ranges': 'bytes', 'Content-Length': '81', 'Cache-Control': 'max-age=86400', 'Expires': 'Sun, 20 Jun 2021 14:39:29 GMT', 'Connection': 'Keep-Alive', 'Content-Type': 'text/html')> http://zhihu.com===Stop . . . <ClientResponse(https://www.zhihu.com/signin?next=/) [200 OK]> <CIMultiDictProxy('Server': 'CLOUD ELB 1.0.0', 'Date': 'Sat, 19 Jun 2021 14:39:29 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Vary': 'Accept-Encoding', 'content-security-policy': "default-src * blob:; img-src * data: blob: resource: t.captcha.qq.com cstaticdun.126.net necaptcha.nosdn.127.net; connect-src * wss: blob: resource:; frame-src 'self' *.zhihu.com mailto: tel: weixin: *.vzuu.com mo.m.taobao.com getpocket.com note.youdao.com safari-extension://com.evernote.safari.clipper-Q79WDW8YH9 zhihujs: captcha.guard.qcloud.com pos.baidu.com dup.baidustatic.com openapi.baidu.com wappass.baidu.com passport.baidu.com *.cme.qcloud.com vs-cdn.tencent-cloud.com t.captcha.qq.com c.dun.163.com; script-src 'self' blob: *.zhihu.com g.alicdn.com qzonestyle.gtimg.cn res.wx.qq.com open.mobile.qq.com 'unsafe-eval' unpkg.zhimg.com unicom.zhimg.com resource: captcha.gtimg.com captcha.guard.qcloud.com pagead2.googlesyndication.com cpro.baidustatic.com pos.baidu.com dup.baidustatic.com i.hao61.net 'nonce-f5ee3225-450b-407f-898c-12f7faa96fb7' hm.baidu.com zz.bdstatic.com b.bdstatic.com imgcache.qq.com vs-cdn.tencent-cloud.com gw.alipayobjects.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net; style-src 'self' 'unsafe-inline' *.zhihu.com unicom.zhimg.com resource: captcha.gtimg.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net", 'x-content-security-policy': "default-src * blob:; img-src * data: blob: resource: t.captcha.qq.com cstaticdun.126.net necaptcha.nosdn.127.net; connect-src * wss: blob: resource:; frame-src 'self' *.zhihu.com mailto: tel: weixin: *.vzuu.com mo.m.taobao.com getpocket.com note.youdao.com safari-extension://com.evernote.safari.clipper-Q79WDW8YH9 zhihujs: captcha.guard.qcloud.com pos.baidu.com dup.baidustatic.com openapi.baidu.com wappass.baidu.com passport.baidu.com *.cme.qcloud.com vs-cdn.tencent-cloud.com t.captcha.qq.com c.dun.163.com; script-src 'self' blob: *.zhihu.com g.alicdn.com qzonestyle.gtimg.cn res.wx.qq.com open.mobile.qq.com 'unsafe-eval' unpkg.zhimg.com unicom.zhimg.com resource: captcha.gtimg.com captcha.guard.qcloud.com pagead2.googlesyndication.com cpro.baidustatic.com pos.baidu.com dup.baidustatic.com i.hao61.net 'nonce-f5ee3225-450b-407f-898c-12f7faa96fb7' hm.baidu.com zz.bdstatic.com b.bdstatic.com imgcache.qq.com vs-cdn.tencent-cloud.com gw.alipayobjects.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net; style-src 'self' 'unsafe-inline' *.zhihu.com unicom.zhimg.com resource: captcha.gtimg.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net", 'x-webkit-csp': "default-src * blob:; img-src * data: blob: resource: t.captcha.qq.com cstaticdun.126.net necaptcha.nosdn.127.net; connect-src * wss: blob: resource:; frame-src 'self' *.zhihu.com mailto: tel: weixin: *.vzuu.com mo.m.taobao.com getpocket.com note.youdao.com safari-extension://com.evernote.safari.clipper-Q79WDW8YH9 zhihujs: captcha.guard.qcloud.com pos.baidu.com dup.baidustatic.com openapi.baidu.com wappass.baidu.com passport.baidu.com *.cme.qcloud.com vs-cdn.tencent-cloud.com t.captcha.qq.com c.dun.163.com; script-src 'self' blob: *.zhihu.com g.alicdn.com qzonestyle.gtimg.cn res.wx.qq.com open.mobile.qq.com 'unsafe-eval' unpkg.zhimg.com unicom.zhimg.com resource: captcha.gtimg.com captcha.guard.qcloud.com pagead2.googlesyndication.com cpro.baidustatic.com pos.baidu.com dup.baidustatic.com i.hao61.net 'nonce-f5ee3225-450b-407f-898c-12f7faa96fb7' hm.baidu.com zz.bdstatic.com b.bdstatic.com imgcache.qq.com vs-cdn.tencent-cloud.com gw.alipayobjects.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net; style-src 'self' 'unsafe-inline' *.zhihu.com unicom.zhimg.com resource: captcha.gtimg.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net", 'x-frame-options': 'SAMEORIGIN', 'strict-transport-security': 'max-age=15552000; includeSubDomains', 'surrogate-control': 'no-store', 'Pragma': 'no-cache', 'Expires': '0', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'X-Backend-Response': '0.027', 'Referrer-Policy': 'no-referrer-when-downgrade', 'X-SecNG-Response': '0.030999898910522', 'x-lb-timing': '0.033', 'x-idc-id': '2', 'Set-Cookie': 'KLBRSID=d017ffedd50a8c265f0e648afe355952|1624113569|1624113569; Path=/', 'Content-Encoding': 'gzip', 'Cache-Control': 'private, must-revalidate, no-cache, no-store, max-age=0', 'Transfer-Encoding': 'chunked', 'X-NWS-LOG-UUID': '4935272354838850546', 'Connection': 'keep-alive', 'X-Cache-Lookup': 'Cache Miss', 'x-edge-timing': '0.063', 'x-cdn-provider': 'tencent')> http://github.com/===Stop . . . Traceback (most recent call last): File "D:/AppData/Python/ForAsync/main.py", line 12, in get_url resp = await client.get(url) File "C:\Users\Haiton\AppData\Local\Programs\Python\Python38\lib\site-packages\aiohttp\client.py", line 544, in _request await resp.start(conn) File "C:\Users\Haiton\AppData\Local\Programs\Python\Python38\lib\site-packages\aiohttp\client_reqrep.py", line 890, in start message, payload = await self._protocol.read() # type: ignore File "C:\Users\Haiton\AppData\Local\Programs\Python\Python38\lib\site-packages\aiohttp\streams.py", line 604, in read await self._waiter aiohttp.client_exceptions.ClientOSError: [WinError 121] 信号灯超时时间已到
可以看到，同一个线程中，并么有挨个执行，而是遇到相应差的和需要IO读写的自动切换了，其中GitHub最先遇到，但是最后执行结束（还是因为超时报错），所以新语法解决协程异步IO是有效的。
上面的POST和GET通过不同的方式实现，是为了展示两种方式，因为aiohttp的 ClientSession 是一个IO的上下文，像文件一样，需要关闭IO，可以像get_url中一样，手动close，也可以像post_url一样，通过with来自动关闭。
