Python中的进程池与线程池

• 引入进程池与线程池

• 使用ProcessPoolExecutor进程池，使用ThreadPoolExecutor

• 使用shutdown

• 使用submit同步调用

• 使用submit异步调用

• 异步+回调函数

• 并发实现套接字通信

引入进程池

在学习线程池之前，我们先看一个例子

# from multiprocessing import Process
# import time
#
# def task(name):
#     print('name',name)
#     time.sleep(1)
# if __name__ == '__main__':
#     start=time.time()
#     p1 = Process(target=task,args=("safly1",))
#     p2 = Process(target=task, args=("safly2",))
#     p3 = Process(target=task, args=("safly3",))
#
#     p1.start()
#     p2.start()
#     p3.start()
#
#     p1.join()
#     p2.join()
#     p3.join()
#
#     print("main")
#
#     end = time.time()
#     print(end- start)

输出如下：

 

 以上的方式是一个个创建进程，这样的耗费时间才1秒多，虽然高效，但是有什么弊端呢？ 
如果并发很大的话，会给服务器带来很大的压力，所以引入了进程池的概念

使用ProcessPoolExecutor进程池

什么时候用池：
池的功能是限制启动的进程数或线程数，
什么时候应该限制？？？
当并发的任务数远远超过了计算机的承受能力时，即无法一次性开启过多的进程数或线程数时
就应该用池的概念将开启的进程数或线程数限制在计算机可承受的范围内

Python3.2开始，标准库为我们提供了concurrent.futures模块，它提供了ThreadPoolExecutor和ProcessPoolExecutor两个类，实现了对threading和multiprocessing的进一步抽象，对编写线程池/进程池提供了直接的支持。

通过ProcessPoolExecutor 来做示例。 
我们来看一个最简单的进程池

from concurrent.futures import ProcessPoolExecutor
import time
def task(name):
    print('name',name)
    time.sleep(1)
if __name__ == '__main__':
    start=time.time()
    p1=ProcessPoolExecutor(2)
    for i in range(5):
        p1.submit(task,i)
    p1.shutdown(wait=True)
    print('主')
    end=time.time()
    print(end-start)

输出如下：

D:\APPS\Python3.7\python.exe "D:/Python/project one/day20180717/进程池与线程池.py"
name 0
name 1
name 2
name 3
name 4
主
3.118098258972168

Process finished with exit code 0

简单解释下： 
ProcessPoolExecutor（2）创建一个进程池，容量为2，循环submit出5个进程，然后就在线程池队列里面，执行多个进程，p1.shutdown(wait=True)意思是进程都执行完毕，在执行主进程的内容

使用shutdown

p1.shutdown(wait=True)是进程池内部的进程都执行完毕，才会关闭，然后执行后续代码 
如果改成false呢？看如下代码

from concurrent.futures import ProcessPoolExecutor
import time
def task(name):
    print('name',name)
    time.sleep(1)
if __name__ == '__main__':
    start=time.time()
    p1=ProcessPoolExecutor(2)
    for i in range(5):
        p1.submit(task,i)
    p1.shutdown(wait=False)
    print('主')
    end=time.time()
    print(end-start)

输出如下：

D:\APPS\Python3.7\python.exe "D:/Python/project one/day20180717/进程池与线程池.py"
主
0.008975744247436523
name 0
name 1
name 2
name 3
name 4

Process finished with exit code 0

使用submit同步调用

同步：提交完任务后就在原地等待，直到任务运行完毕并且拿到返回值后，才运行下一行代码

from concurrent.futures import ProcessPoolExecutor
import time, random, os

def piao(name, n):
    print('%s is piaoing %s' % (name, os.getpid()))
    time.sleep(1)
    return n ** 2


if __name__ == '__main__':
    p = ProcessPoolExecutor(2)
    start = time.time()
    for i in range(5):
        res=p.submit(piao,'safly %s' %i,i).result() #同步调用
        print(res)

    p.shutdown(wait=True)
    print('主', os.getpid())

    stop = time.time()
    print(stop - start)

D:\APPS\Python3.7\python.exe "D:/Python/project one/day20180717/进程池与线程池.py"
safly 0 is piaoing 11448
0
safly 1 is piaoing 11800
1
safly 2 is piaoing 11448
4
safly 3 is piaoing 11800
9
safly 4 is piaoing 11448
16
主 8516
5.095325946807861

Process finished with exit code 0

使用submit异步调用

异步：提交完任务（绑定一个回调函数）后不原地等待，直接运行下一行代码，等到任务运行有返回值自动触发回调的函数的运行

from concurrent.futures import ThreadPoolExecutor
import time
def task(name):
    print('name',name)
    time.sleep(1)
if __name__ == '__main__':
    start=time.time()
    p1=ThreadPoolExecutor(2)
    for i in range(5):
        p1.submit(task,i)
    p1.shutdown(wait=True)
    print('主')
    end=time.time()
    print(end-start)

D:\APPS\Python3.7\python.exe "D:/Python/project one/day20180717/进程池与线程池.py"
name 0
name 1
name 2
name 3
name 4
主
3.003053903579712

使用回调函数+异步

进程

# from concurrent.futures import ProcessPoolExecutor,ThreadPoolExecutor
# import os
# import time
# import random
#
# def task(n):
#     print('%s run...' %os.getpid())
#     time.sleep(5)
#     return n**2
#
# def parse(future):
#     time.sleep(1)
#     res=future.result()
#     print('%s 处理了 %s' %(os.getpid(),res))
#
# if __name__ == '__main__':
#     pool=ProcessPoolExecutor(4)
#     # pool.submit(task,1)
#     # pool.submit(task,2)
#     # pool.submit(task,3)
#     # pool.submit(task,4)
#
#     start=time.time()
#     for i in range(1,5):
#         future=pool.submit(task,i)
#         future.add_done_callback(parse) # parse会在futrue有返回值时立刻触发，并且将future当作参数传给parse
#     pool.shutdown(wait=True)
#     stop=time.time()
#     print('主',os.getpid(),(stop - start))

from concurrent.futures import ProcessPoolExecutor,ThreadPoolExecutor
from threading import current_thread
import os
import time
import random

def task(n):
    print('%s run...' %current_thread().name)
    time.sleep(5)
    return n**2

def parse(future):
    time.sleep(1)
    res=future.result()
    print('%s 处理了 %s' %(current_thread().name,res))

if __name__ == '__main__':
    pool=ThreadPoolExecutor(4)
    start=time.time()
    for i in range(1,5):
        future=pool.submit(task,i)
        future.add_done_callback(parse) # parse会在futrue有返回值时立刻触发，并且将future当作参数传给parse
    pool.shutdown(wait=True)
    stop=time.time()
    print('主',current_thread().name,(stop - start))

并发实现套接字通信

from socket import *
from threading import Thread

def talk(conn):
    while True:
        try:
            data=conn.recv(1024)
            if len(data) == 0:break
            conn.send(data.upper())
        except ConnectionResetError:
            break
    conn.close()

def server(ip,port,backlog=5):
    server = socket(AF_INET, SOCK_STREAM)
    server.bind((ip, port))
    server.listen(backlog)

    print('starting...')
    while True:
        conn, addr = server.accept()

        t = Thread(target=talk, args=(conn,))
        t.start()

if __name__ == '__main__':
    server('127.0.0.1',8080)

from socket import *
import os

client=socket(AF_INET,SOCK_STREAM)
client.connect(('127.0.0.1',8080))

while True:
    msg='%s say hello' %os.getpid()
    client.send(msg.encode('utf-8'))
    data=client.recv(1024)
    print(data.decode('utf-8'))

扩展：

回调函数（callback）是什么？

以下均来自知乎：

 

回调函数（callback）是什么？ - no.body的回答 - 知乎 https://www.zhihu.com/question/19801131/answer/27459821

非常经典的回答加举例。