concurrent

　concurrent.futures

异步并行任务模块

提供两个池执行器

ThreadPoolExcutor 异步调用的线程池Executor

ProcessPoolExecutor 异步调用的进程池Executor

　

 

首先需要定义一个池的执行器对象,Executor类子类的对象

method implication
ThreadPoolExecutor(max_workers=1) 创建一个线程池,最大数量为max_workers个,来同时异步执行,返回Executor实例
submit(fn,*args,**kwargs) 提交执行的函数及其参数,返回Future实例
shutdown(wait=True) 清理池

 

Future类

method implication
result() 查看调用的返回结果
done() 如果调用被成功的取消或执行完成,返回True
cancelled() 调用被成功取消,返回True
running() 正在运行且不能被取消,返回True
cancel() 尝试取消调用,如果已经执行且不能取消返回Fasle,否则返回True
result(timeout=None) 取返回的结果,超时为None,一直等待返回,超时设置到期,抛出concurrent.futures.TimeoutError异常
exception(timeout=None) 取返回的异常,超时为None,一直等待返回,超时设置到期,抛出concurrent,futuresTimeoutError异常

 

复制代码
import threading,multiprocessing,logging,time
from concurrent import futures
FORMAT='%(asctime)-10s\t[%(processName)s:%(threadName)s, %(process)d:%(thread)d] %(message)s'
logging.basicConfig(format=FORMAT,level=logging.DEBUG,datefmt='%H:%M:%S')

def work(b):
    logging.debug('begin to work {}'.format(b))
    logging.info(threading.enumerate())
    time.sleep(5)
    logging.debug('work finished {}'.format(b))

executor=futures.ThreadPoolExecutor(max_workers=3)

fs=[]
for b in range(3):
    f=executor.submit(work,b)
    fs.append(f)

for b in range(3,6):
    f=executor.submit(work,b)
    fs.append(f)

for b in fs:
    print(b,id(b))

while True:
    time.sleep(2)
    logging.warning(threading.enumerate())
    flag=True
    for b in fs:
        flag=flag and b.done()
        logging.debug('b.done(): {}, b: {}'.format(b.done(),b))
    if flag:
        executor.shutdown()
        logging.debug(threading.enumerate())
        break
复制代码

result:

复制代码
13:51:17      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] begin to work 0
13:51:17      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, initial daemon)>]
13:51:17      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] begin to work 1
13:51:17      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] begin to work 2
13:51:17      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:17      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
<Future at 0x17fc4a5c1c0 state=running> 1648271671744
<Future at 0x17fc4a691f0 state=running> 1648271725040
<Future at 0x17fc4a695e0 state=running> 1648271726048
<Future at 0x17fc4a69970 state=pending> 1648271726960
<Future at 0x17fc4a69b20 state=pending> 1648271727392
<Future at 0x17fc4a69c10 state=pending> 1648271727632
13:51:19      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a5c1c0 state=running>
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a691f0 state=running>
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a695e0 state=running>
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69970 state=pending>
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69b20 state=pending>
13:51:19      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69c10 state=pending>
13:51:21      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a5c1c0 state=running>
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a691f0 state=running>
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a695e0 state=running>
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69970 state=pending>
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69b20 state=pending>
13:51:21      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69c10 state=pending>
13:51:22      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] work finished 0
13:51:22      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] begin to work 3
13:51:22      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:22      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] work finished 2
13:51:22      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] begin to work 4
13:51:22      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:22      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] work finished 1
13:51:22      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] begin to work 5
13:51:22      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:23      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a5c1c0 state=finished returned NoneType>
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a691f0 state=finished returned NoneType>
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a695e0 state=finished returned NoneType>
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69970 state=running>
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69b20 state=running>
13:51:23      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69c10 state=running>
13:51:25      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a5c1c0 state=finished returned NoneType>
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a691f0 state=finished returned NoneType>
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a695e0 state=finished returned NoneType>
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69970 state=running>
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69b20 state=running>
13:51:25      [MainProcess:MainThread, 21968:25852] b.done(): False, b: <Future at 0x17fc4a69c10 state=running>
13:51:27      [MainProcess:ThreadPoolExecutor-0_0, 21968:13464] work finished 3
13:51:27      [MainProcess:ThreadPoolExecutor-0_1, 21968:9248] work finished 5
13:51:27      [MainProcess:ThreadPoolExecutor-0_2, 21968:15896] work finished 4
13:51:27      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 13464)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9248)>, <Thread(ThreadPoolExecutor-0_2, started daemon 15896)>]
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a5c1c0 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a691f0 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a695e0 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a69970 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a69b20 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] b.done(): True, b: <Future at 0x17fc4a69c10 state=finished returned NoneType>
13:51:27      [MainProcess:MainThread, 21968:25852] [<_MainThread(MainThread, started 25852)>]

Process finished with exit code 0
View Code
复制代码

 

 

线程池一旦创建了线程对象,线程池中的线程对象不需要频繁清除

 

ProcessPoolExecutor对象:

方法相同,使用多进程

复制代码
import threading,multiprocessing,logging,time
from concurrent import futures
FORMAT='%(asctime)-10s\t[%(processName)s:%(threadName)s, %(process)d:%(thread)d] %(message)s'
logging.basicConfig(format=FORMAT,level=logging.DEBUG,datefmt='%H:%M:%S')

def work(b):
    logging.debug('begin to work {}'.format(b))
    logging.info(multiprocessing.current_process())
    time.sleep(5)
    logging.debug('work finished {}'.format(b))

if __name__ == "__main__":
    executor=futures.ProcessPoolExecutor(max_workers=3)
    fs=[]
    for b in range(3):
        v=executor.submit(work,b) # 提交工作,返回future对象
        fs.append(v)
    for b in range(3,6):
        v=executor.submit(work,b)
        fs.append(v)

    while True:
        time.sleep(2)
        logging.critical(multiprocessing.current_process())
        flag=True
        for b in fs:
            flag=flag and b.done()
            logging.warning('b.done(): {}, b: {}'.format(b.done(),b))
        if flag:
　　　　　　　executor.shutdown() # 清理池
            logging.debug(multiprocessing.current_process())
            break
复制代码

result:

复制代码
C:\Users\zhanggw.YANGCHE\AppData\Local\Programs\Python\Python38\python.exe C:/Users/zhanggw.YANGCHE/PycharmProjects/intern/supervene1.py
14:04:42      [SpawnProcess-1:MainThread, 16904:20212] begin to work 0
14:04:42      [SpawnProcess-1:MainThread, 16904:20212] <SpawnProcess name='SpawnProcess-1' parent=22612 started>
14:04:42      [SpawnProcess-2:MainThread, 23548:17856] begin to work 1
14:04:42      [SpawnProcess-3:MainThread, 9408:26652] begin to work 2
14:04:42      [SpawnProcess-2:MainThread, 23548:17856] <SpawnProcess name='SpawnProcess-2' parent=22612 started>
14:04:42      [SpawnProcess-3:MainThread, 9408:26652] <SpawnProcess name='SpawnProcess-3' parent=22612 started>
14:04:44      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a18dd250 state=running>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a18e54c0 state=running>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901c40 state=running>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901d00 state=running>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901df0 state=pending>
14:04:44      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901ee0 state=pending>
14:04:46      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a18dd250 state=running>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a18e54c0 state=running>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901c40 state=running>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901d00 state=running>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901df0 state=pending>
14:04:46      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901ee0 state=pending>
14:04:47      [SpawnProcess-1:MainThread, 16904:20212] work finished 0
14:04:47      [SpawnProcess-1:MainThread, 16904:20212] begin to work 3
14:04:47      [SpawnProcess-1:MainThread, 16904:20212] <SpawnProcess name='SpawnProcess-1' parent=22612 started>
14:04:47      [SpawnProcess-2:MainThread, 23548:17856] work finished 1
14:04:47      [SpawnProcess-3:MainThread, 9408:26652] work finished 2
14:04:47      [SpawnProcess-2:MainThread, 23548:17856] begin to work 4
14:04:47      [SpawnProcess-3:MainThread, 9408:26652] begin to work 5
14:04:47      [SpawnProcess-2:MainThread, 23548:17856] <SpawnProcess name='SpawnProcess-2' parent=22612 started>
14:04:47      [SpawnProcess-3:MainThread, 9408:26652] <SpawnProcess name='SpawnProcess-3' parent=22612 started>
14:04:48      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18dd250 state=finished returned NoneType>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18e54c0 state=finished returned NoneType>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901c40 state=finished returned NoneType>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901d00 state=running>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901df0 state=running>
14:04:48      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901ee0 state=running>
14:04:50      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18dd250 state=finished returned NoneType>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18e54c0 state=finished returned NoneType>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901c40 state=finished returned NoneType>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901d00 state=running>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901df0 state=running>
14:04:50      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901ee0 state=running>
14:04:52      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18dd250 state=finished returned NoneType>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18e54c0 state=finished returned NoneType>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901c40 state=finished returned NoneType>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901d00 state=running>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901df0 state=running>
14:04:52      [MainProcess:MainThread, 22612:18032] b.done(): False, b: <Future at 0x1b4a1901ee0 state=running>
14:04:52      [SpawnProcess-1:MainThread, 16904:20212] work finished 3
14:04:52      [SpawnProcess-2:MainThread, 23548:17856] work finished 4
14:04:52      [SpawnProcess-3:MainThread, 9408:26652] work finished 5
14:04:54      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18dd250 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a18e54c0 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901c40 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901d00 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901df0 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] b.done(): True, b: <Future at 0x1b4a1901ee0 state=finished returned NoneType>
14:04:54      [MainProcess:MainThread, 22612:18032] <_MainProcess name='MainProcess' parent=None started>

Process finished with exit code 0
View Code
复制代码

 

 

 

 

 

 

concurrent.futures.ThreadPoolExecutor & concurrent.futures.ProcessPoolExecutor 均继承concurrent.futures._base.Executor父类,Executor实现了__enter__() & __exit__() 方法,支持上下文管理,可以使用with语句

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

复制代码
from concurrent import futures
import logging

with futures.ThreadPoolExecutor(max_workers=1) as executor:
    future.submit(pow,2,3)
    logging.warning(future.result())
复制代码

 

 

 

复制代码
from concurrent import futures
import logging

if __name__ == '__main__':
    with futures.ProcessPoolExecutor(max_workers=1) as executor:
        future=executor.submit(pow,3,4)
        logging.warning(future.result())
复制代码

 

 

 

统一了线程池 & 进程池的调用,简化了编程

缺点:无法设置线程名称

posted @   ascertain  阅读(443)  评论(0)  编辑  收藏  举报
编辑推荐：
· 基于Microsoft.Extensions.AI核心库实现RAG应用
· Linux系列：如何用heaptrack跟踪.NET程序的非托管内存泄露
· 开发者必知的日志记录最佳实践
· SQL Server 2025 AI相关能力初探
· Linux系列：如何用 C#调用 C方法造成内存泄露
阅读排行：
· 震惊！C++程序真的从main开始吗？99%的程序员都答错了
· 【硬核科普】Trae如何「偷看」你的代码？零基础破解AI编程运行原理
· 单元测试从入门到精通
· 上周热点回顾（3.3-3.9）
· winform 绘制太阳，地球，月球 运作规律
点击右上角即可分享
微信分享提示
AI IDE Trae