concurrent

　concurrent.futures

异步并行任务模块

提供两个池执行器

ThreadPoolExcutor 异步调用的线程池Executor

ProcessPoolExecutor 异步调用的进程池Executor

首先需要定义一个池的执行器对象,Executor类子类的对象

method	implication
ThreadPoolExecutor(max_workers=1)	创建一个线程池,最大数量为max_workers个,来同时异步执行,返回Executor实例
submit(fn,args,*kwargs)	提交执行的函数及其参数,返回Future实例
shutdown(wait=True)	清理池

Future类

method	implication
result()	查看调用的返回结果
done()	如果调用被成功的取消或执行完成,返回True
cancelled()	调用被成功取消,返回True
running()	正在运行且不能被取消,返回True
cancel()	尝试取消调用,如果已经执行且不能取消返回Fasle,否则返回True
result(timeout=None)	取返回的结果,超时为None,一直等待返回,超时设置到期,抛出concurrent.futures.TimeoutError异常
exception(timeout=None)	取返回的异常,超时为None,一直等待返回,超时设置到期,抛出concurrent,futuresTimeoutError异常

import threading,multiprocessing,logging,time
from concurrent import futures
FORMAT='%(asctime)-10s\t[%(processName)s:%(threadName)s, %(process)d:%(thread)d] %(message)s'
logging.basicConfig(format=FORMAT,level=logging.DEBUG,datefmt='%H:%M:%S')

def work(b):
    logging.debug('begin to work {}'.format(b))
    logging.info(threading.enumerate())
    time.sleep(5)
    logging.debug('work finished {}'.format(b))

executor=futures.ThreadPoolExecutor(max_workers=3)

fs=[]
for b in range(3):
    f=executor.submit(work,b)
    fs.append(f)

for b in range(3,6):
    f=executor.submit(work,b)
    fs.append(f)

for b in fs:
    print(b,id(b))

while True:
    time.sleep(2)
    logging.warning(threading.enumerate())
    flag=True
    for b in fs:
        flag=flag and b.done()
        logging.debug('b.done(): {}, b: {}'.format(b.done(),b))
    if flag:
        executor.shutdown()
        logging.debug(threading.enumerate())
        break

result:

13:51:17      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] begin to work 0
13:51:17      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, initial daemon)>]
13:51:17      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] begin to work 1
13:51:17      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] begin to work 2
13:51:17      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:17      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
<Future at 0x17fc4a5c1c0 state=running> 1648271671744
<Future at 0x17fc4a691f0 state=running> 1648271725040
<Future at 0x17fc4a695e0 state=running> 1648271726048
<Future at 0x17fc4a69970 state=pending> 1648271726960
<Future at 0x17fc4a69b20 state=pending> 1648271727392
<Future at 0x17fc4a69c10 state=pending> 1648271727632
13:51:19      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a5c1c0 state=running>
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a691f0 state=running>
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a695e0 state=running>
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69970 state=pending>
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69b20 state=pending>
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69c10 state=pending>
13:51:21      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a5c1c0 state=running>
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a691f0 state=running>
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a695e0 state=running>
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69970 state=pending>
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69b20 state=pending>
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69c10 state=pending>
13:51:22      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] work finished 0
13:51:22      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] begin to work 3
13:51:22      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:22      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] work finished 2
13:51:22      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] begin to work 4
13:51:22      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:22      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] work finished 1
13:51:22      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] begin to work 5
13:51:22      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:23      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a5c1c0 state=finished returned NoneType>
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a691f0 state=finished returned NoneType>
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a695e0 state=finished returned NoneType>
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69970 state=running>
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69b20 state=running>
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69c10 state=running>
13:51:25      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a5c1c0 state=finished returned NoneType>
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a691f0 state=finished returned NoneType>
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a695e0 state=finished returned NoneType>
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69970 state=running>
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69b20 state=running>
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69c10 state=running>
13:51:27      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] work finished 3
13:51:27      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] work finished 5
13:51:27      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] work finished 4
13:51:27      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a5c1c0 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a691f0 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a695e0 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a69970 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a69b20 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a69c10 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>]

Process finished with exit code 0

View Code

线程池一旦创建了线程对象,线程池中的线程对象不需要频繁清除

ProcessPoolExecutor对象:

方法相同,使用多进程

import threading,multiprocessing,logging,time
from concurrent import futures
FORMAT='%(asctime)-10s\t[%(processName)s:%(threadName)s, %(process)d:%(thread)d] %(message)s'
logging.basicConfig(format=FORMAT,level=logging.DEBUG,datefmt='%H:%M:%S')

def work(b):
    logging.debug('begin to work {}'.format(b))
    logging.info(multiprocessing.current_process())
    time.sleep(5)
    logging.debug('work finished {}'.format(b))

if __name__ == "__main__":
    executor=futures.ProcessPoolExecutor(max_workers=3)
    fs=[]
    for b in range(3):
        v=executor.submit(work,b) # 提交工作,返回future对象
        fs.append(v)
    for b in range(3,6):
        v=executor.submit(work,b)
        fs.append(v)

    while True:
        time.sleep(2)
        logging.critical(multiprocessing.current_process())
        flag=True
        for b in fs:
            flag=flag and b.done()
            logging.warning('b.done(): {}, b: {}'.format(b.done(),b))
        if flag:
　　　　　　　executor.shutdown() # 清理池
            logging.debug(multiprocessing.current_process())
            break

result:

C:\Users\zhanggw.YANGCHE\AppData\Local\Programs\Python\Python38\python.exe C:/Users/zhanggw.YANGCHE/PycharmProjects/intern/supervene1.py
14:04:42      [SpawnProcess-1:MainThread, 16904:20212] begin to work 0
14:04:42      [SpawnProcess-1:MainThread, 16904:20212] <SpawnProcess name='SpawnProcess-1' parent=22612 started>
14:04:42      [SpawnProcess-2:MainThread, 23548:17856] begin to work 1
14:04:42      [SpawnProcess-3:MainThread, 9408:26652] begin to work 2
14:04:42      [SpawnProcess-2:MainThread, 23548:17856] <SpawnProcess name='SpawnProcess-2' parent=22612 started>
14:04:42      [SpawnProcess-3:MainThread, 9408:26652] <SpawnProcess name='SpawnProcess-3' parent=22612 started>
14:04:44      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a18dd250 state=running>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a18e54c0 state=running>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901c40 state=running>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901d00 state=running>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901df0 state=pending>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901ee0 state=pending>
14:04:46      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a18dd250 state=running>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a18e54c0 state=running>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901c40 state=running>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901d00 state=running>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901df0 state=pending>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901ee0 state=pending>
14:04:47      [SpawnProcess-1:MainThread, 16904:20212] work finished 0
14:04:47      [SpawnProcess-1:MainThread, 16904:20212] begin to work 3
14:04:47      [SpawnProcess-1:MainThread, 16904:20212] <SpawnProcess name='SpawnProcess-1' parent=22612 started>
14:04:47      [SpawnProcess-2:MainThread, 23548:17856] work finished 1
14:04:47      [SpawnProcess-3:MainThread, 9408:26652] work finished 2
14:04:47      [SpawnProcess-2:MainThread, 23548:17856] begin to work 4
14:04:47      [SpawnProcess-3:MainThread, 9408:26652] begin to work 5
14:04:47      [SpawnProcess-2:MainThread, 23548:17856] <SpawnProcess name='SpawnProcess-2' parent=22612 started>
14:04:47      [SpawnProcess-3:MainThread, 9408:26652] <SpawnProcess name='SpawnProcess-3' parent=22612 started>
14:04:48      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18dd250 state=finished returned NoneType>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18e54c0 state=finished returned NoneType>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901c40 state=finished returned NoneType>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901d00 state=running>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901df0 state=running>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901ee0 state=running>
14:04:50      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18dd250 state=finished returned NoneType>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18e54c0 state=finished returned NoneType>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901c40 state=finished returned NoneType>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901d00 state=running>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901df0 state=running>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901ee0 state=running>
14:04:52      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18dd250 state=finished returned NoneType>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18e54c0 state=finished returned NoneType>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901c40 state=finished returned NoneType>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901d00 state=running>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901df0 state=running>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901ee0 state=running>
14:04:52      [SpawnProcess-1:MainThread, 16904:20212] work finished 3
14:04:52      [SpawnProcess-2:MainThread, 23548:17856] work finished 4
14:04:52      [SpawnProcess-3:MainThread, 9408:26652] work finished 5
14:04:54      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18dd250 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18e54c0 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901c40 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901d00 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901df0 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901ee0 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>

Process finished with exit code 0

View Code

concurrent.futures.ThreadPoolExecutor & concurrent.futures.ProcessPoolExecutor 均继承concurrent.futures._base.Executor父类,Executor实现了__enter__() & __exit__() 方法,支持上下文管理,可以使用with语句

from concurrent import futures
import logging

with futures.ThreadPoolExecutor(max_workers=1) as executor:
    future.submit(pow,2,3)
    logging.warning(future.result())

from concurrent import futures
import logging

if __name__ == '__main__':
    with futures.ProcessPoolExecutor(max_workers=1) as executor:
        future=executor.submit(pow,3,4)
        logging.warning(future.result())

统一了线程池 & 进程池的调用,简化了编程

缺点:无法设置线程名称

posted @ 2020-10-29 17:12 ascertain 阅读(379) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

ascertain

concurrent

公告