day10---异步I/O,gevent协程
协程
协程,又称微线程,纤程。英文名Coroutine。一句话说明什么是线程:协程是一种用户态的轻量级线程。
协程拥有自己的寄存器上下文和栈。协程调度切换时,将寄存器上下文和栈保存到其他地方,在切回来的时候,恢复先前保存的寄存器上下文和栈。因此:
协程能保留上一次调用时的状态(即所有局部状态的一个特定组合),每次过程重入时,就相当于进入上一次调用的状态,换种说法:进入上一次离开时所处逻辑流的位置。
协程的好处:
- 无需线程上下文切换的开销
- 无需原子操作锁定及同步的开销
- 方便切换控制流,简化编程模型
- 高并发+高扩展性+低成本:一个CPU支持上万的协程都不是问题。所以很适合用于高并发处理。
缺点:
- 无法利用多核资源:协程的本质是个单线程,它不能同时将 单个CPU 的多个核用上,协程需要和进程配合才能运行在多CPU上.当然我们日常所编写的绝大部分应用都没有这个必要,除非是cpu密集型应用。
- 进行阻塞(Blocking)操作(如IO时)会阻塞掉整个程序
使用yield实现协程操作例子
import time import queue def consumer(name): print("--->starting eating baozi...") while True: new_baozi = yield print("[%s] is eating baozi %s" % (name,new_baozi)) #time.sleep(1) def producer(): r = con.__next__() r = con2.__next__() n = 0 while n < 5: n +=1 con.send(n) con2.send(n) print("\033[32;1m[producer]\033[0m is making baozi %s" %n ) if __name__ == '__main__': con = consumer("c1") con2 = consumer("c2") p = producer()
#!/usr/bin/env python #-*- coding:utf-8 -*- # Author:DCC from greenlet import greenlet import gevent ''' def test1(): print(12) gr2.switch() print(34) gr2.switch() def test2(): print(56) gr1.switch() print(78) gr1 = greenlet(test1) gr2 = greenlet(test2) gr1.switch() ''' def foo(): print("Running in foo") gevent.sleep(4) print("switch to foo") def bar(): print("Running in bar") gevent.sleep(1) print("已经调到bar") def func3(): print("Running in func3") gevent.sleep(2) print("func3 again") gevent.joinall([ gevent.spawn(foo), gevent.spawn(bar), gevent.spawn(func3), ])
遇到IO阻塞时会自动切换任务
爬虫
#!/usr/bin/env python #-*- coding:utf-8 -*- # Author:DCC from urllib import request import gevent,time from gevent import monkey monkey.patch_all()#把当前程序的所有的IO操作单独做上标记 #之前 gevent是不是捕获 urllib的IO操作的,所有,无法进行切换 def f(url): print("GET: %s" % url) resp = request.urlopen(url) data = resp.read() print("%d bytes received from %s." % (len(data),url)) urls = [ "https://www.python.org/", "https://www.yahoo.com/", "https://github.com/" ] start_time = time.time() for url in urls: f(url) print("同步cost",time.time()-start_time) async_start_time = time.time() gevent.joinall([ gevent.spawn(f,"https://www.python.org/"), gevent.spawn(f,"https://www.yahoo.com/"), gevent.spawn(f,"https://github.com/"), ]) print("异步cost",time.time()-async_start_time)
通过gevent实现单线程下的多socket并发
server side
import sys import socket import time import gevent from gevent import socket,monkey monkey.patch_all() def server(port): s = socket.socket() s.bind(('0.0.0.0', port)) s.listen(500) while True: cli, addr = s.accept() gevent.spawn(handle_request, cli) def handle_request(conn): try: while True: data = conn.recv(1024) print("recv:", data) conn.send(data) if not data: conn.shutdown(socket.SHUT_WR) except Exception as ex: print(ex) finally: conn.close() if __name__ == '__main__': server(8001)
client side
import socket HOST = 'localhost' # The remote host PORT = 8001 # The same port as used by the server s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((HOST, PORT)) while True: msg = bytes(input(">>:"),encoding="utf8") s.sendall(msg) data = s.recv(1024) #print(data) print('Received', repr(data)) s.close()
记录Python学习过程,给自己改变的机会。