1月12日学习内容整理:爬虫性能相关
@@@老师博客:::
http://www.cnblogs.com/linhaifeng/articles/7806303.html
@@@老师博客内容:::
上述无论哪种解决方案其实没有解决一个性能相关的问题:IO阻塞,无论是多进程还是多线程,在遇到IO阻塞时都会被操作系统强行剥夺走CPU的执行权限,程序的执行效率因此就降低了下来。
解决这一问题的关键在于,我们自己从应用程序级别检测IO阻塞然后切换到我们自己程序的其他任务执行,这样把我们程序的IO降到最低,我们的程序处于就绪态就会增多,以此来迷惑操作系统,操作系统便以为我们的程序是IO比较少的程序,从而会尽可能多的分配CPU给我们,这样也就达到了提升程序执行效率的目的
1、在python3.3之后新增了asyncio模块,可以帮我们检测IO(只能是网络IO),实现应用程序级别的切换
import asyncio @asyncio.coroutine def task(task_id,senconds): print('%s is start' %task_id) yield from asyncio.sleep(senconds) #只能检测网络IO,检测到IO后切换到其他任务执行 print('%s is end' %task_id) tasks=[task(task_id=1,senconds=3),task(task_id=2,senconds=4)] loop=asyncio.get_event_loop() loop.run_until_complete(asyncio.gather(*tasks)) loop.close() 基本使用
2、但asyncio模块只能发tcp级别的请求,不能发http协议,因此,在我们需要发送http请求的时候,需要我们自定义http报头
#我们爬取一个网页的过程,以https://www.python.org/doc/为例,将关键步骤列举如下 #步骤一:向www.python.org这台主机发送tcp三次握手,是IO阻塞操作 #步骤二:封装http协议的报头 #步骤三:发送http协议的请求包,是IO阻塞操作 #步骤四:接收http协议的响应包,是IO阻塞操作 import asyncio @asyncio.coroutine def get_page(host,port=80,url='/'): #步骤一(IO阻塞):发起tcp链接,是阻塞操作,因此需要yield from recv,send=yield from asyncio.open_connection(host,port) #步骤二:封装http协议的报头,因为asyncio模块只能封装并发送tcp包,因此这一步需要我们自己封装http协议的包 requset_headers="""GET %s HTTP/1.0\r\nHost: %s\r\n\r\n""" % (url, host,) # requset_headers="""POST %s HTTP/1.0\r\nHost: %s\r\n\r\nname=egon&password=123""" % (url, host,) requset_headers=requset_headers.encode('utf-8') #步骤三(IO阻塞):发送http请求包 send.write(requset_headers) yield from send.drain() #步骤四(IO阻塞):接收http协议的响应包 text=yield from recv.read() #其他处理 print(host,url,text) send.close() print('-===>') return 1 tasks=[get_page(host='www.python.org',url='/doc'),get_page(host='www.cnblogs.com',url='linhaifeng'),get_page(host='www.openstack.org')] loop=asyncio.get_event_loop() results=loop.run_until_complete(asyncio.gather(*tasks)) loop.close() print('=====>',results) #[1, 1, 1] asyncio+自定义http协议报头
3、自定义http报头多少有点麻烦,于是有了aiohttp模块,专门帮我们封装http报头,然后我们还需要用asyncio检测IO实现切换
import aiohttp import asyncio @asyncio.coroutine def get_page(url): print('GET:%s' %url) response=yield from aiohttp.request('GET',url) data=yield from response.read() print(url,data) response.close() return 1 tasks=[ get_page('https://www.python.org/doc'), get_page('https://www.cnblogs.com/linhaifeng'), get_page('https://www.openstack.org') ] loop=asyncio.get_event_loop() results=loop.run_until_complete(asyncio.gather(*tasks)) loop.close() print('=====>',results) #[1, 1, 1] asyncio+aiohttp
4、此外,还可以将requests.get函数传给asyncio,就能够被检测了
import requests import asyncio @asyncio.coroutine def get_page(func,*args): print('GET:%s' %args[0]) loog=asyncio.get_event_loop() furture=loop.run_in_executor(None,func,*args) response=yield from furture print(response.url,len(response.text)) return 1 tasks=[ get_page(requests.get,'https://www.python.org/doc'), get_page(requests.get,'https://www.cnblogs.com/linhaifeng'), get_page(requests.get,'https://www.openstack.org') ] loop=asyncio.get_event_loop() results=loop.run_until_complete(asyncio.gather(*tasks)) loop.close() print('=====>',results) #[1, 1, 1] asyncio+requests模块的方法
5、还有之前在协程时介绍的gevent模块
from gevent import monkey;monkey.patch_all() import gevent import requests def get_page(url): print('GET:%s' %url) response=requests.get(url) print(url,len(response.text)) return 1 # g1=gevent.spawn(get_page,'https://www.python.org/doc') # g2=gevent.spawn(get_page,'https://www.cnblogs.com/linhaifeng') # g3=gevent.spawn(get_page,'https://www.openstack.org') # gevent.joinall([g1,g2,g3,]) # print(g1.value,g2.value,g3.value) #拿到返回值 #协程池 from gevent.pool import Pool pool=Pool(2) g1=pool.spawn(get_page,'https://www.python.org/doc') g2=pool.spawn(get_page,'https://www.cnblogs.com/linhaifeng') g3=pool.spawn(get_page,'https://www.openstack.org') gevent.joinall([g1,g2,g3,]) print(g1.value,g2.value,g3.value) #拿到返回值 gevent+requests
6、封装了gevent+requests模块的grequests模块
#pip3 install grequests import grequests request_list=[ grequests.get('https://wwww.xxxx.org/doc1'), grequests.get('https://www.cnblogs.com/linhaifeng'), grequests.get('https://www.openstack.org') ] ##### 执行并获取响应列表 ##### # response_list = grequests.map(request_list) # print(response_list) ##### 执行并获取响应列表(处理异常) ##### def exception_handler(request, exception): # print(request,exception) print("%s Request failed" %request.url) response_list = grequests.map(request_list, exception_handler=exception_handler) print(response_list) grequests
7、twisted:是一个网络框架,其中一个功能是发送异步请求,检测IO并自动切换
''' #问题一:error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted pip3 install C:\Users\Administrator\Downloads\Twisted-17.9.0-cp36-cp36m-win_amd64.whl pip3 install twisted #问题二:ModuleNotFoundError: No module named 'win32api' https://sourceforge.net/projects/pywin32/files/pywin32/ #问题三:openssl pip3 install pyopenssl ''' #twisted基本用法 from twisted.web.client import getPage,defer from twisted.internet import reactor def all_done(arg): # print(arg) reactor.stop() def callback(res): print(res) return 1 defer_list=[] urls=[ 'http://www.baidu.com', 'http://www.bing.com', 'https://www.python.org', ] for url in urls: obj=getPage(url.encode('utf=-8'),) obj.addCallback(callback) defer_list.append(obj) defer.DeferredList(defer_list).addBoth(all_done) reactor.run() #twisted的getPage的详细用法 from twisted.internet import reactor from twisted.web.client import getPage import urllib.parse def one_done(arg): print(arg) reactor.stop() post_data = urllib.parse.urlencode({'check_data': 'adf'}) post_data = bytes(post_data, encoding='utf8') headers = {b'Content-Type': b'application/x-www-form-urlencoded'} response = getPage(bytes('http://dig.chouti.com/login', encoding='utf8'), method=bytes('POST', encoding='utf8'), postdata=post_data, cookies={}, headers=headers) response.addBoth(one_done) reactor.run() twisted的用法
8、tornado
from tornado.httpclient import AsyncHTTPClient from tornado.httpclient import HTTPRequest from tornado import ioloop def handle_response(response): """ 处理返回值内容(需要维护计数器,来停止IO循环),调用 ioloop.IOLoop.current().stop() :param response: :return: """ if response.error: print("Error:", response.error) else: print(response.body) def func(): url_list = [ 'http://www.baidu.com', 'http://www.bing.com', ] for url in url_list: print(url) http_client = AsyncHTTPClient() http_client.fetch(HTTPRequest(url), handle_response) ioloop.IOLoop.current().add_callback(func) ioloop.IOLoop.current().start() Tornado