ThreadPoolExecutor

ThreadPoolExecutor ProcessPoolExecutor:

从Python3.2开始，标准库为我们提供了 concurrent.futures 模块，它提供了 ThreadPoolExecutor (线程池)和ProcessPoolExecutor (进程池)两个类。

相比 threading 等模块，该模块通过 submit 返回的是一个 future 对象，它是一个未来可期的对象，通过它可以获悉线程的状态主线程(或进程)中可以获取某一个线程(进程)执行的状态或者某一个任务执行的状态及返回值：

1. 主线程可以获取某一个线程（或者任务的）的状态，以及返回值。

2. 当一个线程完成的时候，主线程能够立即知道。

3. 让多线程和多进程的编码接口一致。

concurrent.Futures模块为异步执行可调用程序提供高级接口。

example1：

from concurrent.futures import ThreadPoolExecutor
import time

# 参数times用来模拟网络请求的时间
def get_html(times):
    time.sleep(times)
    print("get page {}s finished".format(times))
    return times

executor = ThreadPoolExecutor(max_workers=2)
# 通过submit函数提交执行的函数到线程池中，submit函数立即返回，不阻塞
task1 = executor.submit(get_html, (3))
task2 = executor.submit(get_html, (2))
# done方法用于判定某个任务是否完成
print(task1.done())
# cancel方法用于取消某个任务,该任务没有放入线程池中才能取消成功
print(task2.cancel())
time.sleep(4)
print(task1.done())
# result方法可以获取task的执行结果
print(task1.result())

# 执行结果
# False  # 表明task1未执行完成
# False  # 表明task2取消失败，因为已经放入了线程池中
# get page 2s finished
# get page 3s finished
# True  # 由于在get page 3s finished之后才打印，所以此时task1必然完成了
# 3     # 得到task1的任务返回值

• ThreadPoolExecutor构造实例的时候，传入max_workers参数来设置线程池中最多能同时运行的线程数目。
• 使用submit函数来提交线程需要执行的任务（函数名和参数）到线程池中，并返回该任务的句柄（类似于文件、画图），注意submit()不是阻塞的，而是立即返回。
• 通过submit函数返回的任务句柄，能够使用done()方法判断该任务是否结束。上面的例子可以看出，由于任务有2s的延时，在task1提交后立刻判断，task1还未完成，而在延时4s之后判断，task1就完成了。
• 使用cancel()方法可以取消提交的任务，如果任务已经在线程池中运行了，就取消不了。这个例子中，线程池的大小设置为2，任务已经在运行了，所以取消失败。如果改变线程池的大小为1，那么先提交的是task1，task2还在排队等候，这是时候就可以成功取消。
• 使用result()方法可以获取任务的返回值。查看内部代码，发现这个方法是阻塞的。

example2:as_complete

上面虽然提供了判断任务是否结束的方法，但是不能在主线程中一直判断啊。有时候我们是得知某个任务结束了，就去获取结果，而不是一直判断每个任务有没有结束。这是就可以使用as_completed方法一次取出所有任务的结果。

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

# 参数times用来模拟网络请求的时间
def get_html(times):
    time.sleep(times)
    print("get page {}s finished".format(times))
    return times

executor = ThreadPoolExecutor(max_workers=2)
urls = [3, 2, 4] # 并不是真的url
all_task = [executor.submit(get_html, (url)) for url in urls]

for future in as_completed(all_task):
    data = future.result()
    print("in main: get page {}s success".format(data))

# 执行结果
# get page 2s finished
# in main: get page 2s success
# get page 3s finished
# in main: get page 3s success
# get page 4s finished
# in main: get page 4s success

example3:

import concurrent.futures
import urllib.request

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

# Retrieve a single page and report the URL and contents
def load_url(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()

# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its URL
    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        try:
            data = future.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('%r page is %d bytes' % (url, len(data)))

as_completed()方法是一个生成器，在没有任务完成的时候，会阻塞，在有某个任务完成的时候，会yield这个任务，就能执行for循环下面的语句，然后继续阻塞住，循环到所有的任务结束。从结果也可以看出，先完成的任务会先通知主线程。

example4:map

from concurrent.futures import ThreadPoolExecutor
import time

# 参数times用来模拟网络请求的时间
def get_html(times):
    time.sleep(times)
    print("get page {}s finished".format(times))
    return times

executor = ThreadPoolExecutor(max_workers=2)
urls = [3, 2, 4] # 并不是真的url

for data in executor.map(get_html, urls):
    print("in main: get page {}s success".format(data))
# 执行结果
# get page 2s finished
# get page 3s finished
# in main: get page 3s success
# in main: get page 2s success
# get page 4s finished
# in main: get page 4s success

使用map方法，无需提前使用submit方法，map方法与python标准库中的map含义相同，都是将序列中的每个元素都执行同一个函数。上面的代码就是对urls的每个元素都执行get_html函数，并分配各线程池。可以看到执行结果与上面的as_completed方法的结果不同，输出顺序和urls列表的顺序相同，就算2s的任务先执行完成，也会先打印出3s的任务先完成，再打印2s的任务完成

example5:

from concurrent.futures import ThreadPoolExecutor, wait, ALL_COMPLETED, FIRST_COMPLETED
import time

# 参数times用来模拟网络请求的时间
def get_html(times):
    time.sleep(times)
    print("get page {}s finished".format(times))
    return times

executor = ThreadPoolExecutor(max_workers=2)
urls = [3, 2, 4] # 并不是真的url
all_task = [executor.submit(get_html, (url)) for url in urls]
wait(all_task, return_when=ALL_COMPLETED)
print("main")
# 执行结果 
# get page 2s finished
# get page 3s finished
# get page 4s finished
# main

wait方法接收3个参数，等待的任务序列、超时时间以及等待条件。等待条件return_when默认为ALL_COMPLETED，表明要等待所有的任务都结束。可以看到运行结果中，确实是所有任务都完成了，主线程才打印出main。等待条件还可以设置为FIRST_COMPLETED，表示第一个任务完成就停止等待。

源码:

# Copyright 2009 Brian Quinlan. All Rights Reserved.

# Licensed to PSF under a Contributor Agreement.



"""Implements ThreadPoolExecutor."""



__author__ = 'Brian Quinlan (brian@sweetapp.com)'



import atexit

from concurrent.futures import _base

import itertools

import queue

import threading

import weakref

import os



# Workers are created as daemon threads. This is done to allow the interpreter

# to exit when there are still idle threads in a ThreadPoolExecutor's thread

# pool (i.e. shutdown() was not called). However, allowing workers to die with

# the interpreter has two undesirable properties:

#   - The workers would still be running during interpreter shutdown,

#     meaning that they would fail in unpredictable ways.

#   - The workers could be killed while evaluating a work item, which could

#     be bad if the callable being evaluated has external side-effects e.g.

#     writing to a file.

#

# To work around this problem, an exit handler is installed which tells the

# workers to exit when their work queues are empty and then waits until the

# threads finish.



_threads_queues = weakref.WeakKeyDictionary()

_shutdown = False



def _python_exit():

    global _shutdown

    _shutdown = True

    items = list(_threads_queues.items())

    for t, q in items:

        q.put(None)

    for t, q in items:

        t.join()



atexit.register(_python_exit)





class _WorkItem(object):

    def __init__(self, future, fn, args, kwargs):

        self.future = future

        self.fn = fn

        self.args = args

        self.kwargs = kwargs



    def run(self):

        if not self.future.set_running_or_notify_cancel():

            return



        try:

            result = self.fn(*self.args, **self.kwargs)

        except BaseException as exc:

            self.future.set_exception(exc)

            # Break a reference cycle with the exception 'exc'

            self = None

        else:

            self.future.set_result(result)





def _worker(executor_reference, work_queue, initializer, initargs):

    if initializer is not None:

        try:

            initializer(*initargs)

        except BaseException:

            _base.LOGGER.critical('Exception in initializer:', exc_info=True)

            executor = executor_reference()

            if executor is not None:

                executor._initializer_failed()

            return

    try:

        while True:

            work_item = work_queue.get(block=True)

            if work_item is not None:

                work_item.run()

                # Delete references to object. See issue16284

                del work_item

                continue

            executor = executor_reference()

            # Exit if:

            #   - The interpreter is shutting down OR

            #   - The executor that owns the worker has been collected OR

            #   - The executor that owns the worker has been shutdown.

            if _shutdown or executor is None or executor._shutdown:

                # Flag the executor as shutting down as early as possible if it

                # is not gc-ed yet.

                if executor is not None:

                    executor._shutdown = True

                # Notice other workers

                work_queue.put(None)

                return

            del executor

    except BaseException:

        _base.LOGGER.critical('Exception in worker', exc_info=True)





class BrokenThreadPool(_base.BrokenExecutor):

    """

    Raised when a worker thread in a ThreadPoolExecutor failed initializing.

    """





class ThreadPoolExecutor(_base.Executor):



    # Used to assign unique thread names when thread_name_prefix is not supplied.

    _counter = itertools.count().__next__



    def __init__(self, max_workers=None, thread_name_prefix='',

                 initializer=None, initargs=()):

        """Initializes a new ThreadPoolExecutor instance.



        Args:

            max_workers: The maximum number of threads that can be used to

                execute the given calls.

            thread_name_prefix: An optional name prefix to give our threads.

            initializer: A callable used to initialize worker threads.

            initargs: A tuple of arguments to pass to the initializer.

        """

        if max_workers is None:

            # Use this number because ThreadPoolExecutor is often

            # used to overlap I/O instead of CPU work.

            max_workers = (os.cpu_count() or 1) * 5

        if max_workers <= 0:

            raise ValueError("max_workers must be greater than 0")



        if initializer is not None and not callable(initializer):

            raise TypeError("initializer must be a callable")



        self._max_workers = max_workers

        self._work_queue = queue.SimpleQueue()

        self._threads = set()

        self._broken = False

        self._shutdown = False

        self._shutdown_lock = threading.Lock()

        self._thread_name_prefix = (thread_name_prefix or

                                    ("ThreadPoolExecutor-%d" % self._counter()))

        self._initializer = initializer

        self._initargs = initargs



    def submit(*args, **kwargs):

        if len(args) >= 2:

            self, fn, *args = args

        elif not args:

            raise TypeError("descriptor 'submit' of 'ThreadPoolExecutor' object "

                            "needs an argument")

        elif 'fn' in kwargs:

            fn = kwargs.pop('fn')

            self, *args = args

        else:

            raise TypeError('submit expected at least 1 positional argument, '

                            'got %d' % (len(args)-1))



        with self._shutdown_lock:

            if self._broken:

                raise BrokenThreadPool(self._broken)



            if self._shutdown:

                raise RuntimeError('cannot schedule new futures after shutdown')

            if _shutdown:

                raise RuntimeError('cannot schedule new futures after '

                                   'interpreter shutdown')



            f = _base.Future()

            w = _WorkItem(f, fn, args, kwargs)



            self._work_queue.put(w)

            self._adjust_thread_count()

            return f

    submit.__doc__ = _base.Executor.submit.__doc__



    def _adjust_thread_count(self):

        # When the executor gets lost, the weakref callback will wake up

        # the worker threads.

        def weakref_cb(_, q=self._work_queue):

            q.put(None)

        # TODO(bquinlan): Should avoid creating new threads if there are more

        # idle threads than items in the work queue.

        num_threads = len(self._threads)

        if num_threads < self._max_workers:

            thread_name = '%s_%d' % (self._thread_name_prefix or self,

                                     num_threads)

            t = threading.Thread(name=thread_name, target=_worker,

                                 args=(weakref.ref(self, weakref_cb),

                                       self._work_queue,

                                       self._initializer,

                                       self._initargs))

            t.daemon = True

            t.start()

            self._threads.add(t)

            _threads_queues[t] = self._work_queue



    def _initializer_failed(self):

        with self._shutdown_lock:

            self._broken = ('A thread initializer failed, the thread pool '

                            'is not usable anymore')

            # Drain work queue and mark pending futures failed

            while True:

                try:

                    work_item = self._work_queue.get_nowait()

                except queue.Empty:

                    break

                if work_item is not None:

                    work_item.future.set_exception(BrokenThreadPool(self._broken))



    def shutdown(self, wait=True):

        with self._shutdown_lock:

            self._shutdown = True

            self._work_queue.put(None)

        if wait:

            for t in self._threads:

                t.join()

    shutdown.__doc__ = _base.Executor.shutdown.__doc__

cocurrent.future模块中的future的意思是未来对象，可以把它理解为一个在未来完成的操作，这是异步编程的基础 。在线程池submit()之后，返回的就是这个future对象，返回的时候任务并没有完成，但会在将来完成。也可以称之为task的返回容器，这个里面会存储task的结果和状态。那ThreadPoolExecutor内部是如何操作这个对象的呢？

下面简单介绍ThreadPoolExecutor的部分代码：

init方法

 

 

 

 

init方法中主要重要的就是任务队列和线程集合，在其他方法中需要使用到。

 

submit方法

 

submit中有两个重要的对象，_base.Future()和_WorkItem()对象，_WorkItem()对象负责运行任务和对future对象进行设置，最后会将future对象返回，可以看到整个过程是立即返回的，没有阻塞。

adjust_thread_count方法

这个方法的含义很好理解，主要是创建指定的线程数。但是实现上有点难以理解，比如线程执行函数中的weakref.ref，涉及到了弱引用等概念，留待以后理解。

_WorkItem对象

 

 

 

_WorkItem对象的职责就是执行任务和设置结果。这里面主要复杂的还是self.future.set_result(result)。

 

线程执行函数--_worker

这是线程池创建线程时指定的函数入口，主要是从队列中依次取出task执行，但是函数的第一个参数还不是很明白。留待以后。

参考资料：

https://github.com/python/cpython/blob/3.7/Lib/concurrent/futures/thread.py

https://www.jianshu.com/p/b9b3d66aa0be

https://blog.csdn.net/dashoumeixi/article/details/80931681